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Preface 


This  report  documents  the  underlying  methodology  of  a  portfolio-analysis  tool  developed  by 
the  RAND  Corporation  for  the  Missile  Defense  Agency’s  Director  of  Business  Management 
Office  (MDA/DM) .  It  also  serves  as  a  user’s  guide. 

The  report  will  be  of  interest  primarily  to  MDA  officials  and  analysts  who  develop 
and  assess  the  agency’s  programs  of  research,  development,  testing  and  evaluation,  and  de¬ 
ployment  for  their  ability  to  generate  real-world  ballistic-missile  defense  capabilities.  It 
should  also  be  useful  to  some  of  MDA’s  contractors,  as  well  as  to  officials  and  analysts 
throughout  the  Department  of  Defense,  because  portfolio-analysis  methods  and  tools  are 
sorely  needed  for  implementation  of  capabilities-based  planning. 

This  is  the  first  version  of  a  new  tool,  so  questions  and  comments  are  especially  wel¬ 
come  and  should  be  addressed  to  the  project  leader  (pdavis@rand.org)  or  the  principal  devel¬ 
oper  (dreyer@rand.org)  at  RAND’s  Santa  Monica,  CA,  office. 

This  research  was  performed  in  the  Acquisition  and  Technology  Policy  Center  of 
the  RAND  National  Defense  Research  Institute  (NDRI).  NDRI  is  a  federally  funded  re¬ 
search  and  development  center  sponsored  by  the  U.S.  Department  of  Defense.  It  conducts 
research  for  the  Office  of  the  Secretary  of  Defense  (OSD),  the  Joint  Staff,  the  Unified  Com¬ 
batant  Commands,  the  defense  agencies,  the  U.S.  Marine  Corps,  and  the  U.S.  Navy.  For 
more  information  on  the  center,  contact  its  director,  Philip  Anton  (by  e-mail, 
Philip_Anton@rand.org,  or  by  mail  at  RAND,  1 776  Main  Street,  Santa  Monica,  CA, 
90407-2138,  telephone  (310)  393-0411). 

More  information  on  RAND  is  available  at  http://www.rand.org. 
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Summary 


RAND’s  Portfolio-Analysis  Tool  for  Missile  Defense  (PAT-MD)  was  built  to  support  high- 
level  discussion  and  decisionmaking  in  the  Missile  Defense  Agency  (MDA)  by  providing 
summary  portfolio-style  characterizations  of  alternative  investment  options.  These  charac¬ 
terizations  may  involve  projected  capabilities  in  different  missions,  such  as  defense  of  the 
homeland  from  long-range  missile  attacks;  the  balance  of  emphasis  across  missions;  the  man¬ 
aging  of  risks;  and  economic  considerations  such  as  relative  cost-effectiveness.  The  portfolio- 
style  depiction  attempts  to  provide  a  holistic,  top-level  view  across  all  of  these  considerations 
and  is  intended  to  facilitate  discussion  of  program  tradeoffs  and  adjustments.  Equally  impor¬ 
tant,  PAT-MD  and  a  companion  tool,  RAND’s  Capabilities  Analysis  Model  for  Missile  De¬ 
fense  (CAMMD),  make  it  possible  to  “zoom”  to  higher  levels  of  detail  in  order  to  under¬ 
stand  the  basis  of  high-level  characterizations  and  how  they  would  change  if  assumptions  or 
priorities  were  changed. 


Characterizing  Investment  Options  with  PAT-MD 

PAT-MD  is  a  tool  that  can  accommodate  many  different  choices  made  by  the  user.  For  ex¬ 
ample,  a  typical  measure  of  ballistic-missile  defense  system  (BMDS)  capability  is  the  fraction 
of  an  attack  that  would  be  intercepted.  With  PAT-MD,  this  can  be  generated  separately  for 
cases  with  different  numbers  of  attackers  and  different  countermeasure  capabilities.  Such  ca¬ 
pabilities  can  also  be  characterized  separately  for  the  missions  of  homeland  defense  (HD), 
defense  of  friends  and  allies  (DOFA),  and  defense  of  deployed  forces  (DODF).  And,  of 
course,  a  given  investment  option  generates  capabilities  over  time,  so  potential  capability  can 
be  assessed  at  different  nominal  slices  of  time,  such  as  5,  10,  or  15  years  into  the  future. 

In  characterizing  risk,  a  typical  application  of  PAT-MD  may  distinguish  between 
strategic  and  technical/programmatic  risks.  An  investment  program  might  mitigate  the  for¬ 
mer  by  assuring  strategic  adaptiveness — i.e.,  the  ability  to  adapt  to  changes  of  mission  em¬ 
phasis,  the  emergence  of  new  threats,  the  pace  at  which  particular  threats  emerge,  or  positive 
opportunities.  Technical  and  programmatic  risks  may  be  mitigated,  for  example,  by  com¬ 
peting  approaches,  competing  contractors,  and  special  risk-reduction  investments. 

PAT-MD  can  highlight  a  variety  of  budget  considerations,  including  an  investment 
option’s  cost  in  the  next  fiscal  year,  over  the  future  years  defense  program  (FYDP),  or  over 
20  years.  The  costs  might  be  expressed  in  nominal  dollars,  constant  dollars,  or  present-value 
terms. 

A  classic  issue  in  portfolio-style  thinking  is  balance.  Will  a  given  investment  program 
provide  an  appropriate  balance  of  capabilities  across  missions,  one  consistent  with  strategic 
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priorities?  Does  the  program  balance  the  need  to  achieve  effectiveness  with  the  desire  to  re¬ 
duce  technical  and  programmatic  risks?  This  issue  is  particularly  troublesome  for  MDA  be¬ 
cause  the  capabilities  needed  for  effective  defense  are  especially  demanding. 

Finally,  an  important  element  of  portfolio-style  summary  assessments  is  the  provid¬ 
ing  of  measures  of  relative  cost-effectiveness.  Analysts  may  use  PAT-MD  for  marginal  or 
chunky  marginal  analysis  and  may  even  use  a  composite  measure  of  an  option’s  effectiveness 
that  considers  the  various  missions  and  classes  of  risk.  However,  the  philosophy  underlying 
PAT-MD  is  that  decisionmakers  can  best  reason  about  the  various  issues  of  balance  by  seeing 
information  presented  simultaneously  in  various  categories  (e.g.,  implications  of  an  invest¬ 
ment  option  for  mid-term  and  longer-term  capability  for  homeland  defense,  defense  of 
forces  abroad  and  allies,  and  various  types  of  risk).  Rolling  such  information  up  into  a  mea¬ 
sure  of  composite  effectiveness  and  relative  cost-effectiveness  should  be  done,  if  at  all,  only  as 
part  of  summarizing  and  tidying  up  once  issues  are  well  understood. 

Consistent  with  this  philosophy,  PAT-MD  can  support  a  limited  exploratory  analysis 
of  how  robust  composite-effectiveness  and  cost-effectiveness  comparisons  turn  out  to  be. 
In  practice,  this  is  important,  because,  unless  care  is  taken,  such  comparisons  can  be  unduly 
sensitive  to  deeply  buried  mathematical  assumptions. 


Understanding  the  Origins  of  High-Level  Assessments 

Although  it  is  difficult  to  quarrel  with  the  need  for  high-level  summary  assessments  across 
multiple  considerations,  such  summaries  necessarily  are  the  result  of  many  assumptions, 
some  of  which  are  subtle  or  even  insidious.  A  core  feature  of  PAT-MD  is  its  ability  to  zoom 
to  higher  levels  of  detail  as  necessary  to  understand  and  second-guess  summary  judgments.  A 
user  observing  a  top-level  scorecard  may,  for  example,  ask  why  an  option  is  characterized  as 
bad  in  providing  capability  for  homeland  defense.  PAT-MD  can  zoom  to  a  level  of  detail 
that  shows  the  factors  and  assumptions  that  led  to  that  characterization,  allowing  the  user  to 
change  many  of  those  interactively,  as  in  “Oh,  that  ‘requirement’  wasn’t  intended  to  be  quite 
so  rigid.  What  happens  if  it  is  relaxed  slightly?” 

Sometimes,  a  second  or  third  level  of  zoom  is  necessary — even  in  discussions  with 
high-level  decisionmakers — to  achieve  an  adequately  deep  understanding  of  the  issues.  The 
information  needed  typically  is  different  in  character  from  that  in  a  portfolio-style  display.  In 
particular,  it  must  reflect  broad,  parametric  capabilities  analysis  such  as  is  provided  by  the 
capabilities-analysis  tool,  CAMMD.  Because  PAT-MD  and  CAMMD  have  been  designed  to 
work  together,  it  is  easy  to  zoom  from  one  to  the  other,  either  in  real  time  or  by  providing 
the  relevant  CAMMD  displays  as  backups  in  a  briefing. 


Underlying  Methodology 

Tools  for  decision  support  provide  summary  information  abstracted  from  more-detailed  con¬ 
siderations.  The  methods  used  to  abstract  the  information  can  materially  affect  results  and 
impressions  about  those  results,  again  sometimes  in  subtle  or  even  insidious  ways.  It  is  there¬ 
fore  important  for  analysts  to  choose  and  tune  the  methods  appropriately,  and  for  decision¬ 
makers  receiving  related  analyses  to  ask  related  questions.  PAT-MD  provides  five  alternative 
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methods  which  correspond  mathematically  to  alternative  aggregation  functions.  Which 
method  is  appropriate  depends  on  the  analysis  and  context.  In  some  cases,  simple  linear 
weighted  sums,  which  are  used  extensively  in  utility-based  decision-analysis  methods,  are 
adequate.  In  other  cases,  nonlinear  methods  are  needed  to  enforce  the  concept  that  a  system 
with  several  critical  components  will  fail  if  any  of  its  critical  components  fail,  a  situation  that 
arises  frequently  in  capabilities-based  planning.  Thus,  doing  even  better  than  required  on 
one  component  does  not  substitute  for  doing  poorly  on  another,  critical  component.  This 
has  important  implications  for  resource  allocation.  PAT-MD  provides  several  ways  to  reflect 
such  system  effects.  Significantly,  PAT-MD  also  provides  a  straightforward  way  to  test  sensi¬ 
tivity  to  goals  and  thresholds  in  order  to  ensure  that  results  are  not  unduly  sensitive  to  arbi¬ 
trary  assumptions. 
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1.  Introduction 


Background 

Portfolio-Analysis  Tools 

A  portfolio-analysis  tool  enables  a  user  to  compare  and  contrast  investment  options  accord¬ 
ing  to  a  number  of  quantitative  and  qualitative  criteria,  including  various  types  of  risk  and 
cost.  For  strategic  planning,  such  a  tool  can  generate  holistic,  top-down  depictions  of  alterna¬ 
tives  and  their  implications,  perhaps  over  many  years  into  the  future.  For  investment  plan¬ 
ning,  such  a  tool  can  assist  in  “balancing”  programs  and  in  marginal  analysis,  i.e.,  assessing 
where  to  add  or  subtract  the  marginal  dollar.  Such  actions  are  essential  in  capabilities-based 
planning.1 

The  Portfolio-Analysis  Tool  for  Missile  Defense  (PAT-MD) 

PAT-MD  is  a  specialized  version  of  an  application-independent  portfolio-analysis  tool  (PAT) 
tuned  for  the  particular  needs  of  the  Missile  Defense  Agency  (MDA).  For  example,  it 
includes  special  input  tables  for  building-block  concepts  that  MDA  refers  to  as  engagement 
sequence  groups  (ESGs).  Also,  investment  options  input  to  PAT-MD  are  structured  to  be 
consistent  with  MDA’s  budget  data,  and  PAT-MD’s  output  displays  have  been  designed 
with  the  director  of  MDA  and  his  senior  staff  in  mind.  Finally,  PAT-MD  was  designed  to 
work  seamlessly  with  RAND’s  Capabilities  Analysis  Model  for  Missile  Defense  (CAMMD), 
a  capabilities-analysis  model  developed  in  parallel  with  it  (Willis,  Bonomo,  Davis,  and 
Fiillestad,  2005). 

PAT-MD  has  been  implemented  in  Microsoft  EXCEL®,  an  application  that  is  ubiq¬ 
uitous  in  the  analytical  world.  We  have  used  it  in  a  cross-platform,  networked  environment 
comprising  both  Windows  and  Macintosh  computers.2  Although  an  on-the-shelf  version  of 
the  generic  PAT  has  not  yet  been  created,  doing  so  should  be  straightforward. 


1  Our  portfolio-management  approach  to  defense  planning  was  first  suggested  in  an  issue  paper  that  discusses  how  the  ap¬ 
proach  compares  to  portfolio  analysis  in  the  financial  world  (Davis,  Gompert,  and  Kugler,  1996).  We  initially  applied  our 
approach  using  the  DynaRank  portfolio  tool  (Hillestad  and  Davis,  1998)  to  assess  alternative  defense-planning  options, 
including  those  embracing  what  has  come  to  be  called  transformation  (Davis,  Kugler,  and  Hillestad,  1997).  Such  portfolio 
methods  should  play  a  key  role  in  analysis  for  capabilities-based  planning  (Davis,  2002). 

2  Our  experience  is  limited  to  reasonably  modern  versions  of  software:  Windows  2000,  Macintosh  OS  10.3,  and  Microsoft 
Office  versions  2000  and  2004. 
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Functionality 

PAT-MD  is  not  a  model;  rather,  it  is  a  tool  that  facilitates  capabilities-based  planning  by  pre¬ 
senting  information  in  a  way  that  is  useful  to  senior  leaders.  It  is  an  empty  vessel,  but  one 

with  many  useful  features: 

1.  Summary  scorecards:  PAT-MD  generates  stoplight  charts,  simple  color-scorecard  summa¬ 
ries  of  how  options  rate  on  a  number  of  juxtaposed  criteria,  including  measures  of  capa¬ 
bilities,  risks,  and  costs.  These  criteria  may  be  quantitative  or  qualitative,  objective  or 
subjective. 

2.  Zooming:  PAT-MD  generates  its  summaries  from  more-detailed  considerations,  which 
can  be  viewed  by  zooming  in  to  a  level  that  provides  a  terse  logic  and  a  measure  of  rigor, 
even  for  qualitative  assessments.  Assumptions  may  or  may  not  be  valid,  of  course,  which 
implies  the  need  to  vary  them. 

3.  Sensitivity  analysis  and  exploratory  analysis:  PAT-MD  allows  the  analyst  to  recognize 
key  assumptions  quickly  and  to  change  them  interactively.  This  may  be  done 
parameter-by-parameter  (e.g.,  by  changing  the  assumed  performance  of  a  particular  inter¬ 
ceptor),  or  it  may  be  done  more  broadly,  by  invoking  a  different  set  of  criteria  and  as¬ 
sumptions  that  represents  a  different  perspective ,  such  as  greater  concern  about  a  future 
stressful  threat  than  about  a  mid-term  modest  threat. 

4.  Alternative  aggregation  methods:  PAT-MD  allows  the  analyst  to  quickly  change  the  charac¬ 
ter  of  the  method  by  which  summary  depictions  are  generated  from  details  (i.e.,  aggre¬ 
gated  from  the  details  or  rolled  up).  The  method  may  involve  simple  linear  weighted 
sums,  or  it  may  be  nonlinear,  assessing  an  investment  option  harshly  if  any  critical  com¬ 
ponent  of  capability  fails  to  achieve  required  performance.  PAT-MD  provides  five  differ¬ 
ent,  useful  aggregation  methods  (discussed  in  Chapter  2) . 

5.  Qnks  to  capability  analysis  and  other  sources  of  data:  PAT-MD  links  to  even  more-detailed 
information,  such  as  that  of  a  capabilities  model,  empirical  data,  or  structured  judg¬ 
ments.  In  practice,  PAT-MD  is  intended  to  be  used  together  with  CAMMD  (Willis  et 
ah,  2005),  so  that  a  user  can  shift  interactively  between  them  while  working  or  while  in  a 
group  discussion. 

6.  V arginal  analysis:  Although  PAT-MD  emphasizes  multi-objective  scorecards,  it  can  also 
generate  aggregate  scores  of  effectiveness  or  cost-effectiveness.  These  can  be  used  for  mar¬ 
ginal  or  chunky  marginal  analysis  of  how  to  spend  (or  cut)  the  next  increment  of  funds. 
Such  work  should  always  include  variation  of  both  assumptions  and  aggregation  meth¬ 
ods.  The  premium  is  on  finding  robustly  sound  choices  rather  than  allegedly  optimal 
strategies  that  are  sensitively  dependent  on  assumptions. 

7.  Facilitated  operations:  At  a  more  mechanical  level,  PAT-MD  automates  a  great  many  te¬ 
dious  spreadsheet  operations,  enabling  users  to  generate  and  manipulate  portfolio-style 
scorecards  and  underlying  detailed  material  quickly.  It  also  provides  a  variety  of  graphics 
to  assist  in  visualizing  the  capabilities  of  the  investment  options. 

Chapters  2  through  5  describe  PAT-MD,  using  notional  examples  and  data  for  the 

missile-defense  problem  in  the  context  of  MDA. 
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Organization  of  the  Report 

Chapter  2  introduces  the  principal  concepts  and  terms  in  PAT-MD  and  presents  a  schematic 
overview.  Chapters  3  and  4  describe  PAT-MD’s  input  and  outputs  in  user-manual  detail. 
Chapter  5  discusses  selected  methodological  issues,  especially  aggregation  methods,  in  more 
detail,  along  with  methods  for  marginal  and  chunky  marginal  analysis.  Chapter  6  presents 
some  concluding  observations,  including  suggestions  and  cautions  for  users  and  some  chal¬ 
lenges  for  future  work.  Appendix  A  summarizes  features  of  PAT-MD  and  compares  them 
with  those  of  DynaRank  (Hillestad  and  Davis,  1998).  Appendix  B  shows  how  to  develop  a 
portfolio  view  from  the  PAT-MD  template.  Users  may  wish  to  use  the  template  and  Appen¬ 
dix  B  early,  especially  if  they  prefer  to  learn  by  doing. 


2.  Overview  of  PAT-MD 


The  Black-Box  Perspective 


From  a  black-box  perspective,  PAT-MD  takes  a  series  of  inputs  and  generates  outputs  in  the 
form  of  portfolio-style  tables  and  various  charts  and  graphics  (Figure  2.1). 

Many  of  the  inputs,  such  as  the  investment  options  to  be  compared,  are  what  one 
might  expect.  For  our  purposes,  an  investment  option  specifies  the  investment  in  each 
budget  category  for  each  year  covered  by  the  analysis.  It  will  include  investments  in  the  de¬ 
velopment  and  deployment  of  the  many  components  of  the  ballistic-missile  defense  system 
(BMDS),  e.g.,  particular  interceptors,  radars,  and  battle-management  systems.  An  invest¬ 
ment  option  will  also  include  year-by-year  investments  in  basic  research  and  development 
(R&D)  and  in  the  general  infrastructure  associated  with  MDA’s  efforts.  Investment  options 
may  differ,  for  example,  in  the  components  developed  and  the  speed  with  which  they  are  de¬ 
veloped,  in  what  will  be  deployed  operationally,  and  so  on.  Investment  options  may  differ 
simply  because  of  cost  considerations,  or  they  may  reflect  alternative  technical  architectures 
or  differences  in  mission  priority,  such  as  defending  the  U.S.  homeland  rather  than  defend¬ 
ing  allies  or  U.S.  forces  deployed  abroad. 

As  shown  in  Figure  2.1,  inputs  to  PAT-MD  also  include  capabilities,  risks,  and  costs 
for  each  investment  option,  as  well  as  control  parameters.  These  determine  the  form  of  the 
outputs,  the  assumptions  and  methods  used  for  evaluation  and  aggregation,  and  so  forth. 
Because  they  can  strongly  affect  how  the  various  options  stack  up  in  summary  displays,  it  is 
important  to  understand  them  and  to  vary  them  systematically  before  drawing  conclusions. 
This  is  discussed  further  below. 

At  this  point,  we  shall  introduce  a  number  of  terms  and  illustrate  some  basic  con¬ 
cepts.  A  more  complete  and  rigorous  treatment  is  presented  in  Chapter  5. 

Figure  2.1 

PAT-MD  as  a  Black  Box 

Investment  options 
Capability  assessments 
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Fundamental  Terms  and  Concepts  in  PAT-MD 

The  Baseline 

Each  investment  option  is  evaluated  in  PAT-MD  on  a  number  of  criteria  called  measures , 
each  of  which  may  be  determined  by  multiple  submeasures.  Submeasures  have  raw  values 
specified  in  the  input  data  for  each  investment  option.  Usually,  PAT-MD  maps  these  raw 
values  into  scores  by  comparing  them  with  goals.1  These  scores  indicate  “goodness,”  or  utility. 
Submeasure  scores  are  combined  to  generate  the  scores  of  measures,  which  are  then  com¬ 
bined  to  generate  a  composite  score  called  effectiveness.  Each  of  these  steps  is  mere  mathe¬ 
matics,  but  each  involves  important  assumptions.  We  can  illustrate  the  basics  with  a  simple, 
contrived  example,  shown  in  Figure  2.2. 

In  this  example,  each  investment  option  is  evaluated  in  PAT-MD  by  only  two  mea¬ 
sures  relating  to  capability  and  also  to  cost  (e.g.,  nominal  dollars  over  a  six-year  future  years 
defense  program  (FYDP)  period  or  over  20  years).  The  option’s  goodness  is  measured  by 
how  well  its  defenses  could  stop  a  small  attack  (Measure  A)  and  how  large  an  attack  could  be 
engaged  before  the  defense  was  saturated  (Measure  B).  Goodness  also  depends  on  cost.  Both 
measures  are  assessed  for  two  cases,  a  simple  attacker  and  one  with  substantial  countermea¬ 
sures.  Success  in  these  two  cases  is  represented  as  submeasure  data.  As  indicated  at  the  bot¬ 
tom  of  Figure  2.2,  as  one  moves  upward,  the  aggregations  are  assumed  to  be  simple  averages. 

Figure  2.2 

An  Example  Illustrating  Measures,  Submeasures,  and  Combining  Rules 
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Relative 

effectiveness 


Effectiveness 


Cost 


Measure  A 

(fraction  of  attackers  stopped 
assuming  no  saturation) 


Measure  B 

(attack  size  that  saturates 
the  defense) 


Submeasure  A.1 
(no  countermeasures) 


Submeasure  A.2 
(countermeasures) 


t 


Submeasure  B.1 
(no  countermeasures) 


Submeasure  B.2 
(countermeasures) 


Assumptions: 

Each  measure's  score  is  the  simple  average  of  its  submeasure  scores. 
Effectiveness  is  the  simple  average  of  Measure  A  and  Measure  B  scores. 
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1  PAT-MD  allows  the  user  to  adopt  any  of  five  different  methods  for  characterizing  options.  One  method,  the  Rankings 
method,  generates  rankings  rather  than  scores  (see  Chapter  5). 
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To  continue  with  the  example,  assume  that  the  data  of  Table  2.1  apply  for  a  par¬ 
ticular  investment  option,  Option  1 .  The  goal  scores  are  user  input  (an  example  of  control 
settings).  The  assumed  goal  is  for  a  defense  that  can  stop  95  percent  of  a  small  attack  that  has 
no  countermeasures  (A.  1)  or  80  percent  of  a  similar  attack  that  does  involve  countermeasures 
(A.2).  Another  goal  is  for  the  defense  to  be  able  to  handle  attacks  of  30  missiles  without  satu¬ 
ration  effects,  whether  or  not  the  attack  includes  countermeasures  (Submeasures  B.l  and 
B.2).  With  this  background,  the  assumed  data  for  Option  1  (i.e.,  the  raw  submeasure  values) 
are  95  percent,  76  percent,  30  missiles,  and  13  missiles.  These  data  might  have  come  from 
the  capabilities  model  (CAMMD)  or  from  a  source  inside  MDA.  The  bottom  row  of  Table 
2.1  shows  the  scores  of  Option  1  as  calculated  by  PAT-MD.  The  score  for  Submeasures  A.  1 
and  A.2  are  1.0  (i.e.,  95/95)  and  0.95  (76/80),  respectively.  The  scores  for  Submeasures  B.l 
and  B.2  are  1.0  (30/30)  and  0.43  (13/30),  respectively.  Given  those  submeasure  scores,  the 
measure-level  scores — assumed  in  the  example  to  be  simple  averages — are  0.98  for  A  and 
0.72  for  B.  Composite  effectiveness,  E,  is  simply  the  average  of  these,  or  0.84.  Table  2.2 
shows  how  cost-effectiveness  is  calculated.  This  example  assumes  two  options,  the  second  of 
which  is  superior.  Option  2’s  effectiveness,  calculated  in  the  same  way  as  that  for  Option  1  in 
Table  2.1,  is  0.95.  Options  1  and  2  cost  $10  billion/year  and  $8  billion/year,  respectively.  In 
this  case,  the  relative  effectiveness  for  Options  1  and  2  is  0.88  and  1.0,  respectively,  because 
Option  2  is  better  and  Option  1  is  compared  to  it  (0.84/0.95  =  0.88).  Cost-effectiveness  is 
defined  as  effectiveness  divided  by  cost  (i.e.,  0.84/10  =  0.084  and  0.95/8  =  0.12).  The  rela¬ 
tive  cost-effectiveness  of  an  option  is  its  cost-effectiveness  divided  by  that  of  the  option  with 
the  greatest  cost-effectiveness.  In  this  case,  the  relative  cost-effectiveness  of  Option  1  is  0.70. 


Table  2.1 

Data  for  One  Investment  Option 


Submeasure  Submeasure 
A.1  A.2 

Measure 

A 

Submeasure 

B.l 

Submeasure  Measure 
B.2  B 

Effectiveness 

Goal 

Stop  95%  of  Stop  80%  of 
attack  attack 

30 

30 

Raw  values  for 
Option  1 

95%  76% 

30 

13 

Scores  for 
Option  1 

1.0  0.95 

0.98 

1.0 

0.43  0.72 

0.84 

Table  2.2 

Cost-Effectiveness  Calculations 

Option 

Cost 

Effectiveness  ($  billions) 

Relative 

Effectiveness 

Cost- 

Effectiveness 

Relative  Cost- 
Effectiveness 

1 

0.84 

10 

0.88 

0.70 

2 

0.95 

8 

1.0 

1.0 

The  Need  to  Generalize 

The  example  conveys  the  basic  ideas,  but  we  must  now  consider  how  many  generalizations 
might  be  needed. 
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1.  Threshold  effects.  It  is  often  desirable  to  reflect  thresholds,  where  a  given  score  is  zero  un¬ 
less  a  threshold  is  reached.  That  would  correspond  to  the  standard  practice  of  “failing”  an 
activity  if  it  doesn’t  reach  some  requirement  or  acceptable  level  of  performance.  Mathe¬ 
matically,  reflecting  thresholds  introduces  another  simple  nonlinearity. 

2.  Different  importances  or  priorities.  Since  submeasures  often  represent  low-level  objectives 
with  different  priorities,  a  measure-level  score  might  be  calculated  as  a  weighted  linear 
sum  of  the  measure’s  submeasure  scores,  rather  than  a  simple  average.  Similarly,  effec¬ 
tiveness  might  be  a  weighted  linear  sum  of  the  measure-level  scores. 

3.  System  effects.  Since  linear  sums  imply  substitutability,  scores  for  systems  with  individual 
critical  components — i.e.,  components  that  must  separately  meet  performance  require¬ 
ments — should  be  calculated  with  nonlinear  combining  rules,  such  as  those  discussed  in 
Chapter  5. 

4.  Other  measures  of  cost  for  cost-effectiveness.  What  is  cost?  The  cost  denominator  used  in 
cost-effectiveness  might,  for  example,  be  a  one-year  cost,  a  six-year  FYDP  cost,  or  a  20- 
year  (life-cycle)  cost.  It  might  be  expressed  in  nominal  or  real  dollars.  It  might  or  might 
not  be  in  discounted  dollars  (i.e.,  in  present- value  terms).  It  might  consist  of  R&D  costs, 
deployment  costs,  or  both.  Various  combinations  could  also  be  used,  such  as  a  weighted 
average  of  one-year  costs  for  the  next  fiscal  year,  for  the  next  six  years,  or  for  20  years. 

The  primary  point  is  that  what  may  at  first  seem  straightforward — providing  a  cal¬ 
culation  engine  for  making  multi-criteria  assessments  and  cost-effectiveness  compari¬ 
sons — turns  out  to  be  complex  in  practice. 


Scoring  and  Aggregation  Methods 

To  provide  flexibility  in  dealing  with  the  issues  noted  above  (possible  generalizations),  PAT- 
MD  has  five  built-in  methods  for  establishing  scores  and  aggregating  upward.  Ideally,  only 
one  such  method  would  be  needed,  thus  simplifying  the  analysis.  However,  theory  and  expe¬ 
rience  tell  us  that  alternative  methods  are  needed — perhaps  even  more  than  those  we  have 
included  in  this  version  of  PAT-MD. 

Three  of  the  five  are  our  core  methods,  which  we  recommend  for  most  MDA  appli¬ 
cations.  Outlined  briefly  in  Figure  2.3  and  Table  2.3,  they  all  include  the  concepts  of  thresh¬ 
olds,  goals,  and  nonlinearity.  Chapter  5  provides  details  on  all  five  methods. 

PAT-MD  Core  Methods  Using  Thresholds,  Goals,  and  Nonlinearity 

Table  2.3  shows  how  submeasure  and  measure  scores  are  calculated  for  the  three  core  meth¬ 
ods  ( Thresholds ,  Weakest  Link,  and  Weak  Thresholds )  and  how  the  composite  score  ( effective¬ 
ness )  is  calculated  from  measure  scores  (the  scores  of  the  various  measures).  First,  raw  sub¬ 
measure  values  are  mapped  into  submeasure  scores,  as  shown  graphically  in  Figure  2.3.  The 
mapping  is  linear  except  that  the  score  is  zero  for  raw  values  below  a  threshold  and  constant 
for  raw  values  above  the  goal.  This  is  a  simple  approximation  of  an  S-curve  function,  as  used 
in  earlier  RAND  work  (Hillestad  and  Davis,  1998).  Note  that  the  curve  can  be  reversed  if 
the  submeasure  represents  a  parameter  for  which  less  is  better  (e.g.,  cost,  expected  number  of 
leakers). 
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Figure  2.3 

Mapping  of  Raw  Submeasure  Values  into  Scores 


Submeasure  score 
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Table  2.3 

Core  Aggregation  Methods  in  PAT-MD 


Method 

Submeasure 

Scores 

Measure  Scores 

Composite 

Effectiveness 

Comment 

Thresholds 

See  Figure  2.3 

0  if  any  raw  value  does  not 
reach  threshold;  other¬ 
wise,  a  weighted  sum  of 
submeasure  scores 

Weighted  sum  of 
measures'  scores 

Appropriate  if  submea¬ 
sures  represent  critical 
components  of  capabil¬ 
ity 

Weakest  Link3 

See  Figure  2.3 

Minimum  of  submeasure 

scores 

Minimum  of  measure  Appropriate  if  both 
scores  submeasures  and 

measures  are  individ¬ 
ually  critical 

Weak  Thresholds 

See  Figure  2.3 

Weighted  sum  of  sub¬ 
measure  scores 

Weighted  sum  of 
measures'  scores 

Appropriate  if  submea¬ 
sures  are  not  all  critical 
and  measures  are  not 
all  critical 

aTh is  method  was  introduced  in  DynaRank  (Hillestad  and  Davis,  1998). 


Next,  measure-level  scores  are  calculated  from  submeasure  scores.  Here  the  methods 
differ  (Table  2.3).  The  default  method  (Thresholds)  characterizes  the  measure-level  score  as 
zero  if  any  of  the  raw  values  do  not  reach  the  threshold  value  defined  for  each  submeasure. 
This  is  intended  to  enforce  the  concept  that  a  system  fails  if  any  of  its  critical  components 
fail;  doing  better  on  one  component  does  not  substitute  for  failing  on  another.2  This  method 


2  This  emphasis  on  mission-system  analysis  (Davis,  2002)  is  consistent  with  thinking  in  terms  of  what  the  Department  of 
Defense  (DoD)  sometimes  calls  mission-capability  packages  (MCPs)  (Alberts,  Garstka,  and  Stein,  1999).  Operational 
commanders  routinely  try  to  identify  all  of  the  critical  enablers  of  their  proposed  operation.  In  contrast,  standard  decision- 
analysis  methods  typically  employ  linear-weighted-sum  techniques,  which  implicitly  evaluate  systems  as  though  further 
improving  one  component  of  a  system  can  substitute  for  improving  another. 
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is  appropriate  if  the  submeasures  do  happen  to  be  individually  critical.  The  Weakest  Link 
method  is  similar  but  even  more  stringent  in  some  respects.  It  assesses  measure-level  score  to 
be  the  lowest  of  the  measure’s  submeasure  scores  and  it  assesses  composite  effectiveness  as  the 
lowest  of  the  measure  scores.  The  third  method,  Weak  Thresholds,  is  less  draconian  than  the 
others.  Both  measure-level  scores  and  composite  effectiveness  are  simply  weighted  sums.  A 
submeasure  that  does  not  reach  the  threshold  value  will  contribute  zero  to  its  parent  measure 
score,  but  the  measure  score  will  not  be  zero  (unless  all  submeasure  scores  are  zero).  This 
method  is  suitable  if  the  submeasures  or  the  measures  are  not  individually  critical. 

These  methods  are  essentially  heuristic  techniques,  and  no  claims  are  made  about 
achieving  “optimal”  decisions  (see  also  Chapter  5).  Evaluating  something  with  rigid  thresh¬ 
olds  or  saying  that  a  system  with  several  components  has  zero  value  if  even  one  of  its  compo¬ 
nents  is  inadequate  simplifies  reality.  In  practice,  these  are  effective  heuristics,  well  under¬ 
stood  by  decisionmakers.  As  an  example,  a  tough  military  commander  may  consider  that  one 
of  his  units  “failed  inspection,”  even  though  the  unit  did  rather  well  in  many  respects.  An 
operational  commander  in  charge  of  a  ballistic-missile  defense  system  (BMDS)  would  regard 
the  system  as  operationally  useless  if  its  kill  vehicles  did  not  work,  even  if  the  system’s  radars, 
infrared  sensors,  missiles,  and  battle-management  capabilities  were  superb. 

As  discussed  in  Chapter  5,  other  threshold-related  possibilities  exist  for  aggregations. 
Nevertheless,  we  believe  that  the  set  of  threshold-related  alternatives  provided  is  adequate; 
however,  this  will  be  reassessed  after  more  experience  is  gained  with  PAT-MD. 

Alternative  Methods  That  Do  Not  Involve  Thresholds 

PAT-MD  includes  two  other  aggregation  methods  that  can  be  useful  in  certain  cases.  Both 
are  defined  more  fully  in  Chapter  5.  The  Goals  method  calculates  a  measure’s  score  as  simply 
the  weighted  fraction  of  the  submeasure  goals  achieved  by  the  option.  Here,  composite  effec¬ 
tiveness  is  a  weighted  linear  sum  of  measure  effectiveness.  The  Rankings  method  focuses  on  a 
relative  ranking  of  investment  options,  rather  than  on  a  comparison  of  scores.  It  uses  a  Borda 
count  method  that  may  be  familiar  to  readers  from  other  domains,  such  as  voting  or  sports 
polls. 


Navigation,  Inputs,  and  Outputs  in  PAT-MD 

This  section  provides  an  overview  of  how  one  navigates  within  PAT-MD,  the  inputs  that 
must  be  provided,  and  the  outputs  it  generates.  Subsequent  chapters  will  describe  inputs  and 
outputs  in  much  more  detail. 

Architecture  and  Navigation 

As  mentioned  earlier,  PAT-MD  is  a  spreadsheet  tool  built  in  EXCEL,  which  provides  a 
spreadsheet-paradigm  means  of  entering  data  and  generates  many  of  the  graphs  and  charts 
needed  for  the  portfolio  tool.  Underlying  the  visible  aspects  of  PAT-MD  is  a  great  deal  of 
Visual  Basic  code,  which  enables  many  operations  that  would  not  be  available  in  an  ordinary 
EXCEL  spreadsheet.  For  example,  the  user  can  modify  some  of  the  names  in  output  displays, 
and  the  changes  will  be  automatically  recorded  in  the  underlying  data  so  that  the  next  view¬ 
ing  of  the  tool  will  show  the  new  names.  Or  the  user  can  modify  the  range  of  years  to  be  in- 
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eluded  in  a  display  by  modifying  that  range  on  the  output  sheet.  PAT-MD  then  can  extract 
the  relevant  data  from  that  label  change  and  correct  the  underlying  control  variable. 

EXCEL  uses  multiple  worksheets  as  input  and  output.  The  user  navigates  among 
sheets  by  clicking  tabs  or  buttons  in  the  currently  visible  sheet.  Figure  2.4  is  a  screenshot 
showing  the  bottom  of  a  PAT-MD  display  and  some  of  its  tabs  (there  are  other  tabs  to  the 
right,  reached  by  clicking  on  the  appropriate  arrows).  In  this  figure,  the  Summary  worksheet 
is  selected. 

Table  2.4  itemizes  PAT-MD’s  sheets,  each  of  which  has  a  tab.  They  are  listed  as  out¬ 
put  sheets  or  input  sheets,  although  there  is  some  cross-functionality.  That  is,  one  can  make 
some  changes  to  input  by  changing  what  is  basically  an  output  sheet. 

Outputs 

With  this  background,  we  now  review  schematically  the  key  outputs  from  PAT-MD  to  pro¬ 
vide  a  general  sense  of  what  PAT-MD  does.  Figure  2.5  shows  our  topmost  display,  the 
Summary  sheet.  This  schematic  suppresses  many  details  that  will  be  discussed  in  Chapter  4. 

Rows  in  the  Summary  sheet  contain  investment  options.  Each  option  is  scored  in  a 
variety  of  ways  represented  by  the  different  columns.  As  shown  in  Figure  2.5,  the  first  block 
of  assessments  is  the  color-coded  measure-level  summary  table.  Blocks  to  the  right  contain 
numeric  values,  such  as  selected  submeasures  that  the  analyst  wishes  to  highlight,  cost  data, 
or  effectiveness  and  cost-effectiveness  values. 

The  essential  challenge  in  working  with  such  high-level  depictions  is  to  assure  that 
they  use  the  right  representation  of  issues  (i.e.,  they  highlight  the  right  considerations)  and 
that  they  reflect  assessments  based  on  solid,  reproducible  analysis.  The  analysis  may  be  based 
on  information  from  sources  such  as  capability  models  or  on  structured  estimates  of  subjec¬ 
tive  considerations  such  as  program  risk.  Or  it  may  reflect  higher-level  subjective  judgments. 

Figure  2.4 

Some  PAT-MD  Worksheet  Tabs 
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Table  2.4 

PAT-MD  Output  and  Input  Sheets 
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Selected  Details 
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Template  Builder 
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Figure  2.5 

Schematic  of  Summary  Sheet 
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PAT-MD  users  should  have  the  ability  to  understand  why  a  given  cell  of  the 
measure-summary  display  is  red  rather  than  green  (bad  rather  than  good)  or  where  various 
numbers  came  from.  This  can  be  done  to  a  significant  extent  by  moving — for  a  particular 
measure — to  the  Details  sheet,  which  is  a  detailed  view  explaining  how  the  top-level  assess¬ 
ment  has  been  generated  from  subordinate  considerations.  Figure  2.6  shows  this  schemati¬ 
cally.  Like  the  Summary  sheet  rows,  those  of  the  Details  sheet  contain  different  investment 


Figure  2.6 

Schematic  of  Details  Sheet 
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options,  and  the  columns  contain  data  on  the  submeasures  that  constitute  a  top-level  mea¬ 
sure.  Truly  understanding  the  assessments  will  sometimes  require  going  into  even  more  de¬ 
tail.  This  may  involve,  for  example,  studying  results  of  exploratory  analyses  with  CAMMD. 

PAT-MD  also  generates  numerous  graphics,  which  can  be  quite  useful  in  the  course 
of  analysis  and  the  presentation  of  results.  Figure  2.7  illustrates  the  range  of  such  graphics, 
and  examples  will  be  discussed  more  fully  in  later  chapters. 

Figure  2.7 

Sample  Output  Displays  from  PAT-MD 
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3.  PAT-MD  Input  Sheets 


This  chapter  describes  each  PAT-MD  worksheet  and  shows  either  a  screen  capture  or  a 
schematic  of  each.  The  figures  have  letters  overlaid  indicating  particular  items  that  we  refer 
to  in  the  discussion.  Names  of  the  worksheets  are  in  italics  throughout  the  chapter. 


Data  Entry  Sheet 

PAT-MD  data  are  categorized  by  measures  and  submeasures,  which  usually  pertain  to  the 
anticipated  capability  or  risk  associated  with  an  investment  option.  Measures  are  entered  as 
headers  of  columns  (A  in  Figure  3.1),  and  the  names  of  the  investment  options  (B)  are  the 
headers  of  the  rows.  Names  of  measures,  submeasures,  and  investment  options  must  be  con¬ 
sistent  across  all  sheets;  if  they  are  not,  errors  will  arise  during  the  generation  of  the  output 
sheets.  The  error  messages  highlight  the  measure  or  investment  option  name  that  is  at  fault, 
enabling  the  user  to  fix  the  problem.  Appendix  B  describes  a  Template  Builder  sheet  that 
populates  the  input  sheets  with  a  consistent  set  of  measure  and  investment-option  names, 
thereby  reducing  or  eliminating  errors  due  to  label  inconsistencies. 

Figure  3.1  presents  a  subset  of  the  columns;  the  full  Data  Entry  sheet  extends  right- 
ward  as  far  as  necessary.  Submeasures  are  shown  for  HD  Potential-2010  and  HD  Potential- 
2020,  the  homeland  defense  potential  in  2010  and  2020.  Measures  are  aggregations  of  one 
or  more  submeasures  (C),  such  as  the  probability  of  engagement  success  (more  properly,  the 
probability  that  a  missile  fired  at  the  United  States  would  be  intercepted),  the  number  of  lay¬ 
ers  (boost,  midcourse,  and  terminal)  in  which  the  incoming  missile  is  engaged,  and  resistance 
to  countermeasures.  The  submeasures  can  be  different  for  different  measures. 

Much  of  the  Data  Entry  sheet  is  filled  in  with  submeasure  values  for  the  investment 
options  and  measures.  These  data  may  come  from  a  capabilities  model  such  as  CAMMD  or 
from  other  sources.  The  rows  at  the  top  of  the  sheet  specify  the  control  parameters  needed  to 
aggregate  from  submeasure  scores  to  measure  scores  and  from  measure  scores  to  composite 
scores  (effectiveness).  How  and  whether  a  given  control  parameter  is  actually  used  depends 
on  the  scoring  and  aggregation  methods  used. 

Scoring  and  Aggregation-Function  Control  Parameters 

The  scoring  and  aggregation  functions  depend  on  parameters  such  as  goal  values.  Four  rows 
in  the  Data  Entry  sheet  are  used  to  set  those  parameters: 
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Figure  3.1 
Data  Entry  Sheet 
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1.  Weight  of  submeasure  in  scoring  functions  (0  to  1)  (D).  This  row  sets  the  relative  weight 
of  the  submeasure  when  calculating  the  measure  score  given  in  the  Summary  sheet’s 
measure-summary  table.  A  weight  of  zero  means  that  the  submeasure  is  not  considered.1 
Thus,  a  submeasure  can  be  built  in  but  then  toggled  on  or  off  as  appropriate  for  a  par¬ 
ticular  analysis.  This  parameter  should  not  be  interpreted  to  specify  relative  weight  across 
measures.  It  provides  only  the  relative  weights  of  submeasures  within  a  given  measure. 
The  relative  weights  of  different  measures  for  further  aggregation  calculations  are  speci¬ 
fied  in  the  Measure  Comments  and  Weights  sheet  described  below. 

2.  High  or  low  values  desired?  (E).  Users  may  wish  to  input  some  measures  or  submeasures 
for  which  more  is  better  and  some  for  which  more  is  worse.  PAT-MD  allows  this,  but  the 
user  must  specify.  This  row  of  the  Data  Entry  table  takes  two  possible  values:  high  and 
low.  For  example,  if  the  submeasure  is  defended  area,  the  appropriate  value  would  be 
high,  because  defending  more  area  is  good.  If  the  submeasure  is  the  desired  date  of  initial 
operating  capability  (IOC),  low  values  would  be  good  (the  sooner  the  better). 

3.  Threshold  value  (F).  This  is  the  threshold  value  described  earlier  for  the  Thresholds, 
Weakest  Link,  and  Weak  Thresholds  scoring  methods. 

4.  Goal  value  (G).  This  is  the  goal  value  used  in  all  of  the  non-rankings-based  scoring  func¬ 
tions.  For  submeasures  where  high  values  are  desired,  the  goal  value  cannot  be  lower  than 
the  threshold  value;  for  submeasures  where  low  values  are  desired,  it  cannot  be  higher 
than  the  threshold  value. 


1  In  this  case,  under  the  Goals  method,  a  measure  will  not  fail  merely  because  a  zero-weight  submeasure  fails. 
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Control  Parameters  for  Cost-Effectiveness 

Effectiveness  and  cost-effectiveness  calculations  aggregate  across  measures  and  must  operate 
on  a  common  scale.  Two  control  parameters  specify  how  to  accomplish  this.  Because  the 
values  of  the  submeasures  are  on  different  scales,  the  two  remaining  control  parameters  set 
up  a  common  scale  of  effectiveness  across  submeasures: 

1.  Submeasure  score  for  threshold  value  (0  to  1)  (H).  This  is  the  score  assigned  to  a  given 
submeasure  if  its  threshold  is  just  reached  (for  scoring  methods  that  incorporate  thresh¬ 
olds). 

2.  Submeasure  score  for  goal  value  (0  to  1)  (I).  This  is  the  score  assigned  to  a  given  sub¬ 
measure  if  its  goal  value  is  reached  or  exceeded.  For  scoring  methods  that  incorporate 
thresholds,  the  effectiveness  score  for  a  value  that  falls  between  the  threshold  value  and 
the  goal  value  is  interpolated,  depending  on  how  close  the  value  is  to  the  goal.  It  is  good 
practice  to  set  this  value  to  1  for  all  submeasures  and  then  weight  the  relative  contribu¬ 
tions  of  each  submeasure  using  the  submeasure-weights  row  above. 

Risk 

In  addition  to  the  submeasures  discussed  so  far,  each  measure  may  include  a  special  sub¬ 
measure  called  Risk  (J).  When  this  submeasure  is  included  and  there  is  text  in  the  cell  de¬ 
scribing  the  risk  of  an  investment  option  with  regard  to  that  measure,  a  flag  appears  on  the 
Summary  sheet,  highlighting  the  risk  of  that  particular  investment  option  for  that  measure. 
The  user  can  view  the  text  in  the  risk  column  on  the  Summary  sheet  by  placing  the  mouse 
cursor  over  the  cell  containing  the  flag. 

Other  Controls 

The  Data  Entry  sheet  is  the  principal  input  sheet.  However,  for  convenience  in  interactive 
work,  many  of  the  items  may  be  viewed  and  modified  on  the  Details  sheet,  which  is  nomi¬ 
nally  an  output  sheet  and  is  described  in  Chapter  4.  If  changes  are  made  there,  however,  they 
will  not  take  effect  until  the  user  clicks  the  Modify  Summary  button  (K)  on  the  Data  Entry 
sheet  or  an  equivalent  button  on  the  Summary  or  Details  sheet. 


Cost  Data  Sheet 

To  incorporate  cost  data  into  the  analysis,  the  user  first  identifies  the  set  of  BMDS  compo¬ 
nents  that  are  under  consideration.  In  addition,  the  user  may  include  as  “components”  catch¬ 
all  investment  categories  covering,  for  example,  low-level  R&D  and  continuing  general  sys¬ 
tems  engineering.  Each  investment  option,  e.g.,  Baseline  (A  in  Figure  3.2),  is  defined  by  the 
amount  of  investment  over  time  (which  we  call  investment  streams)  for  each  BMDS  compo¬ 
nent  (B).  Each  row  defines  an  investment  stream  for  one  component,  and  Figure  3.2  shows  a 
partial  list  of  components,  along  with  fictitious  budgets,  to  illustrate  how  this  sheet  func¬ 
tions.  These  investment  streams  are  broken  down  by  year  (C)  and  by  the  amount  of  money 
allocated  to  research,  development,  testing,  and  evaluation  (RDT&E)  and  the  deployment  of 
the  BMDS.  A  single  column  separates  the  tables  for  RDT&E  and  deployment  costs  for  each 
investment  option  (D),  with  a  single  row  separating  consecutive  investment  options  (E). 
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Figure  3.2 
Cost  Data  Sheet 
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Clicking  the  Export  Cost  Data  button  (F)  produces  two  text  files  that  can  be  exported  into 
CAMMD:  RDTECost.txt  and  DeplCost.txt. 

These  cost  data  are  used  in  PAT-MD  in  cost-effectiveness  calculations  and  for  de¬ 
termining  the  initial  operating  capability  (IOC)  dates  for  both  system  components  and 
ESGs.  Each  system  component  is  considered  available  for  deployment  when  the  cumulative 
investment  in  it  passes  a  user-defined  cost  threshold  (these  thresholds  appear  on  the  ESG 
Data  sheet  described  next).  An  ESG  is  considered  available  for  deployment  when  all  of  the 
components  it  comprises  are  available.  Thus,  the  IOC  of  an  ESG  is  the  maximum  of  the 
IOCs  of  its  components. 

The  cost  data  are  presented  in  a  variety  of  ways  on  the  Summary  and  ESG  Table  out¬ 
put  sheets.  As  explained  in  Chapter  4,  users  can  present  RDT&E,  deployment,  or  total  costs 
on  the  output  sheets  for  any  sequence  of  years. 


ESG  Data  Sheet 

The  ESG  concept  is  very  important  in  current  MDA  thinking.  ESGs,  however,  are  not  typi¬ 
cally  the  basis  for  budget  line  items,  nor  are  they  acquired  and  deployed.  Moreover,  some 
components  that  are  acquired  and  deployed,  such  as  the  Defense  Support  Program’s  early- 
warning  satellites  or  Aegis-based  radars,  could  be  parts  of  several  ESGs.  The  ESG  Data  sheet, 
shown  in  Figure  3.3,  specifies  which  components  constitute  ESGs  of  interest. 

For  an  ESG  to  be  effective,  all  of  the  critical  components  it  comprises  must  be  fully 
developed.  This  is  a  core  element  of  system  planning.  The  main  piece  of  the  ESG  Data  sheet 
is  a  table  with  the  system  components  as  the  rows  (A)  and  the  various  ESGs  under  considera¬ 
tion  as  the  columns  (B).  If  a  component  is  a  critical  part  of  an  ESG,  there  is  a  1  in  the  corre¬ 
sponding  cell  in  the  table  (C).  The  column  at  the  left  of  the  table  gives  the  amount  of  money 
that  needs  to  be  spent  for  a  system  to  be  fully  developed  (D),  as  well  as  a  set  of  three  classifi- 
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Figure  3.3 
ESG  Data  Sheet 
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cation  categories  to  enable  tracking  on  the  ESG  Table  output  sheet  of  the  amount  of  mon- 
eyspent  in  each  category  for  each  investment  option  (E-G).  The  categorization  in  this  exam¬ 
ple  is  by  whether  a  system  is  involved  in  boost-phase,  midcourse,  or  terminal  intercept,  but 
the  user  can  change  this.  A  system  can  appear  in  more  than  one  category. 


Perspective  Cases  Sheet 

The  results  on  the  Summary  and  Details  sheets  are  driven  by  a  particular  set  of  assumptions 
about  the  relative  weights  of  the  measures  and  submeasures,  as  well  as  the  goal  and  threshold 
values  set  for  each  submeasure.  These  values  can  be  set  directly  on  the  Data  Entry  sheet. 
However,  one  of  the  primary  problems  with  scoring  methods  that  depend  on  aggregation  is 
that  the  results  depend  upon  the  rules  used  for  aggregating  (e.g.,  the  scoring  methods,  the 
relative  weights,  thresholds).  As  first  discussed  some  years  ago  (Hillestad  and  Davis,  1998),  a 
better  way  to  proceed  is  often  to  define  a  set  of  alternative  perspectives  that  span  the  space  of 
interesting  disagreements.  Each  such  perspective  has  its  own  assumptions  about  scoring 
methods,  weights,  and  so  on.  Given  a  set  of  perspectives,  one  can  do  exploratory  analysis 
across  that  set.  If  a  given  investment  option  ranks  highly  across  different  perspectives,  it  can 
be  considered  an  especially  good  candidate.  Interestingly,  this  often  occurs  because  the  cold 
reality  of  costs  dominates  the  calculations,  and  people  with  very  different  perspectives  may 
find  themselves  agreeing,  albeit  grudgingly,  on  the  merits  of  a  particular  alternative  (Davis, 
2002). 

The  Perspective  Cases  sheet  (a  portion  of  which  is  shown  in  Figure  3.4)  contains  the 
perspectives  available  for  exploration.  After  it  is  filled  out,  the  user  may  shift  among  perspec¬ 
tives  when  viewing  the  Summary  sheet  and  may  generate  comparison  displays.  The  user  may 
specify  alternative  perspectives  on  the  sheet  or  may  change  some  of  the  parameters  of  an  ex¬ 
isting  perspective  in  the  Data  Entry  sheet,  in  which  case  he  will  have  the  opportunity  to  save 
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Figure  3.4 

Portion  of  Perspective  Cases  Sheet 
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the  revised  set  of  parameters  as  a  new  perspective,  which  will  be  entered  into  this  sheet  auto¬ 
matically. 

To  add  a  perspective  directly  onto  the  Perspective  Cases  sheet,  one  inserts  a  row  with 
the  name  of  the  perspective  case  (for  example,  “Emphasis  on  Measure  1  ”)  in  the  first  column 
below  the  top  eight  rows,  which  contain  all  of  the  measure  and  submeasure  names,  weights, 
goal  values,  and  threshold  values.  This  row  must  also  have,  for  every  measure,  at  least  one 
column  with  a  number  representing  the  weight  of  the  measure  (defined  on  the  Measure 
Comments  and  Weights  sheet).  (See  Appendix  B  for  a  worked-out  example.)  In  Figure  3.4,  the 
weights  on  Measure  1  and  Measure  2  are  2  and  1  for  the  “Emphasis  on  Measure  1”  perspec¬ 
tive,  and  they  are  reversed  for  the  “Emphasis  on  Measure  2”  perspective. 

When  a  new  perspective  is  selected  on  the  Summary  sheet,  the  first  rows  on  the  Data 
Entry  sheet  are  automatically  replaced  by  the  corresponding  rows  associated  with  the  perspec¬ 
tive  on  the  Perspective  Cases  sheet,  and  the  weights  of  the  measures  are  updated  on  the  Mea¬ 
sure  Comments  and  Weights  sheet.  PAT-MD  checks  to  make  sure  the  measures  and  sub¬ 
measures  on  the  Perspective  Cases  sheet  match  up  column-for-column  with  the  corresponding 
columns  on  the  Data  Entry  sheet. 


Measure  Comments  and  Weights  Sheet 

All  the  measures  are  listed  in  the  first  column  of  the  Measure  Comments  and  Weights  sheet 
(shown  in  Figure  3.5).  Their  names  must  be  typed  precisely  as  in  the  other  sheets  on  which 
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Figure  3.5 

Measure  Comments  and  Weights  Sheet 
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they  appear.  The  next  column  contains  comments  describing  the  measures  that  appear  on 
the  Summary  sheet  when  the  user  passes  the  cursor  over  a  measure’s  name.  Finally,  the  third 
column  is  used  to  specify  the  relative  weight  of  each  measure  for  effectiveness  calculations.2 


IOC  to  Cost  Data  Sheet 

In  standard  use,  investment  options  are  the  PAT-MD  inputs,  and  those  options  specify 
yearly  expenditures  in  each  BMDS  component  over  a  time  period.  One  of  PAT-MD’s  out¬ 
puts  (informed  by  other  sources,  such  as  CAMMD)  is  potential  capability  versus  time.  Deci¬ 
sionmakers,  however,  may  wish  to  invert  this  and  specify  the  dates  by  which  to  have  IOC  of 
every  ESC.  In  that  case,  the  investment  streams  needed  to  achieve  these  goals  is  an  output. 
PAT-MD  can  generate  first-cut  approximations  to  these  investment  streams,  based  on  the 
current  investment  schedule  for  each  system  component.  This  is  intended  as  an  alternative 
means  of  generating  investment  options,  but  the  output  from  the  IOC  to  Cost  Data  sheet 
(sketched  in  Figure  3.6)  is  completely  independent  of  the  rest  of  the  tool  unless  the  invest¬ 
ment  options  are  entered  in  the  Cost  Data  sheet  and  elsewhere.  The  raw  values  in  the  Data 
Entry  sheet  for  these  investment  options  would  still  have  to  be  generated  by  some  other 
means,  as  well. 

The  IOC  to  Cost  Data  sheet  has  the  current  investment  schedule  for  each  system 
component.  If  a  component’s  schedule  needs  to  be  accelerated  or  decelerated,  the  investment 
schedule  must  be  compressed  or  stretched  accordingly.  For  example,  assume  a  system  was 
originally  scheduled  to  be  completed  in  six  years,  with  investments  of  $150  million  for  each 
of  the  first  four  years  and  $200  million  for  each  of  the  last  two  years.  If  the  schedule  were 


2  The  weights  can  also  be  specified  on  the  Summary  sheet  (nominally  an  output  sheet)  or  changed  when  the  perspective  is 
changed.  The  changes  are  automatically  entered  on  the  Measure  Comments  and  Weights  sheet  when  the  appropriate  update 
button  is  clicked. 
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Figure  3.6 

IOC  to  Cost  Data  Sheet 
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accelerated  to  achieve  completion  in  four  years,  each  year  in  the  new  schedule  would  repre¬ 
sent  1.5  years  of  investment  from  the  old  schedule.  Thus,  the  new  investment  schedule 
would  be  $225  million  for  each  of  the  first  two  years  (Year  1  in  the  new  schedule  would  have 
all  of  the  investment  from  Year  1  in  the  old  schedule  plus  half  of  the  investment  from  Year  2 
in  the  old  schedule),  $250  million  for  Year  3  (all  of  Year  4  and  half  of  Year  5  from  the  old 
schedule),  and  $300  million  for  Year  4  (half  of  Year  5  and  all  of  Year  6  from  the  old  sched¬ 
ule).  This  is  only  a  first  approximation  of  how  the  money  would  be  reallocated,  but  it  is  a 
good  starting  point.  Finally,  because  stretching  or  accelerating  programs  creates  instabilities 
that  typically  raise  overall  costs,  we  include  a  multiplier  to  estimate  this  effect  roughly.  It  is 
set  by  using  the  Cost  Multiplier  dropdown  menu  on  the  IOC  to  Cost  Data  sheet.  The  multi¬ 
plier  may  be  set  to  1,  assuming  that  there  is  no  effect,  but  effects  of  5  to  30  percent  would 
not  be  surprising.  Thus,  the  multiplier  should  be  informed  by  historical  experience. 

Below  the  cost  streams  on  the  IOC  to  Cost  Data  sheet  is  a  table  with  the  investment 
options  as  the  rows  and  all  of  the  ESGs  as  the  columns.  Each  entry  in  the  table  corresponds 
to  the  desired  ESC  availability  date  for  a  particular  investment  option.  When  the  Generate 
Cost  Data  button  is  selected,  the  RDT&E  cost  data  for  each  investment  option  appear  on 
the  IOC  to  Cost  Output  sheet.  Note  that  the  deployment  funds  must  be  entered  separately. 
That  is,  PAT-MD  does  not  calculate  a  deployment  schedule. 


Dropdown  Sheet 


The  entries  of  all  of  the  dropdown  menus  are  stored  on  the  Dropdown  sheet.  These  menus 
are  generated  on  the  fly  by  PAT-MD,  so  there  is  no  reason  for  the  user  to  ever  modify  this 
sheet.  Indeed,  doing  so  would  cause  problems,  and  PAT-MD  has  not  yet  been  fully  “gorilla- 
proofed.” 


4.  PAT-MD  Output  Sheets 


Summary  Sheet 

The  Summary  sheet  is  the  main  output  for  PAT-MD.  Its  overall  structure  was  shown  in  Fig¬ 
ure  2.5.  It  is  shown  here  in  Figures  4.1a  and  4.1b,  and  its  components  are  discussed  below. 
For  each  investment  option,  there  are  three  sets  or  tables  of  results: 

•  Nine  columns  of  color-coded  measure-summary  assessments  (Figure  4.1a) 

•  Four  columns  displaying  numeric  data  (e.g.,  cost  data  or  the  raw  values  of  important 
submeasures)  (three  columns  are  shown  in  Figure  4.1b) 

•  Six  columns  of  cost  data  (three  of  which  are  displayed),  with  two  additional  columns 
presenting  the  relative  effectiveness  and  relative  cost-effectiveness  of  the  investment 
options  (Figure  4.1b). 

To  include  a  measure  in  the  measure-summary  portion  of  the  sheet  (the  stoplight 
chart),  one  can  either  use  the  template  to  generate  the  Summary  sheet  (Appendix  B)  or  enter 
the  name  of  the  measure  (A  in  Figure  4.1a)  in  any  of  the  entries  on  the  first  row  above  the 
first  table.  This  name  must  match  the  name  of  a  measure  listed  on  the  Data  Entry  sheet.  To 
include  a  submeasure  in  the  numeric  data  table,  one  enters  the  name  of  the  submeasure  using 
the  format  measure:: submeasure  (A  in  Figure  4.1b).  There  are  three  types  of  cost  metrics  to  be 
included  in  the  cost-data  columns:  RDT&E  costs  only,  deployment  costs  only,  and  total 
costs  (RDT&E  +  deployment)  (B  in  Figure  4.1b).  The  format  for  the  titles  of  those  columns 
is  xxxx  cost:  yyyy  -  zzzz  ($M),  where  xxxx  =  Total,  RDT&E,  or  Deployment,  andyyyy  and 
zzzz  are  the  beginning  and  ending  years  of  the  time  period  under  consideration.  Alterna¬ 
tively,  the  column  may  be  titled  xxxx  Cost:  yyyy  ($M)  if  only  one  year  of  expenditures  is  un¬ 
der  consideration. 

The  two  final  columns  are  called  Relative  Effectiveness  (C  in  Figure  4.1b)  and  Rela¬ 
tive  Cost-Effectiveness  (D).  The  scoring  method  used  for  the  effectiveness  calculation  is  se¬ 
lected  from  the  Cost-Effectiveness  Scoring  Method  dropdown  menu  (E),  and  the  cost  metric 
used  in  the  cost-effectiveness  calculation  is  selected  using  the  Cost-Effectiveness  Cost  Metric 
dropdown  menu  (F).  The  cost  metric  may  be  any  of  the  six  cost  data  columns  on  the  Sum¬ 
mary  sheet,  or  RDT&E,  deployment,  or  total  costs  over  the  entire  time  period  under  consid¬ 
eration.  In  addition,  a  discount  rate  can  be  applied  to  the  cost  numbers,  using  the  Discount 
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Figure  4.1a 
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Figure  4.1b 

Right  Side  of  Summary  Sheet 
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Rate  dropdown  menu  (J).  The  effectiveness  and  cost-effectiveness  values  are  scaled  so  that 
the  topmost  investment  option  has  a  value  of  1  and  all  other  investment  options  are  com¬ 
pared  to  it.1 

The  colors  in  the  measure-summary  table  (the  stoplight  chart)  depend  on  the  scoring 
method  chosen  by  using  the  Scoring  Method  dropdown  menu  (C  in  Figure  4.1a) — Goals, 


1  Another  approach  would  be  to  compare  to  some  baseline  or  some  ideal,  but  that  did  not  seem  appropriate  for  missile- 
defense  options. 
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Thresholds,  Weak  Thresholds,  Weakest  Link,  or  Rankings.  The  meaning  of  each  color  in  the 
table  appears  below  the  table  (B  in  Figure  4.1a).  If  the  scoring  method  is  changed,  the  sheet 
will  be  automatically  regenerated  under  the  new  scoring  method. 

The  rows  of  the  table  may  be  sorted  by  the  values  in  any  column,  including  Relative 
Effectiveness  and  Relative  Cost-Effectiveness.  To  sort  the  rows,  the  user  clicks  on  the  Sorting 
Column  (D  in  Figure  4.1a)  or  Sorting  Method  (E)  dropdown  menu  to  change  the  desired 
sorting  column  or  method  (that  is,  in  increasing  or  decreasing  order).  The  sheet  will  be 
automatically  regenerated  with  the  rows  sorted.  Alternatively,  clicking  on  the  Sort  Rows  but¬ 
ton  (F)  will  sort  the  rows  and  regenerate  the  sheet.  In  addition,  the  user  can  select  a  different 
perspective  (see  the  Perspective  Cases  sheet  in  Chapter  3),  using  the  Perspective  dropdown 
menu  (G). 

Above  each  of  the  measure-summary  columns  is  a  button  called  Detail  (FI  in  Figure 
4.1a).  Clicking  this  button  brings  up  the  Details  sheet  for  the  particular  measure  (column). 
Similarly,  clicking  the  Related  Details  button  (G  in  Figure  4.1b)  for  any  of  the  numeric  data 
columns  brings  up  the  details  of  every  submeasure  that  shares  the  same  measure  as  the  sub¬ 
measure  in  that  column.  Clicking  the  Cost  Detail  button  (FI  in  Figure  4.1b)  generates  the 
tables  and  charts  on  the  ESG  Table  worksheet,  described  below. 

Every  investment  option  has  a  comment  (I  in  Figure  4.1a)  in  the  cell  containing  its 
name  (J)  (indicated  by  a  red  triangle  in  the  upper  right-hand  corner  of  the  cell).  Passing  the 
mouse  over  that  cell  brings  up  a  comment  that  has  information  on  which  ESGs  are  devel¬ 
oped  under  that  investment  option,  as  well  as  when  each  ESG  becomes  available.  Similarly,  if 
a  cell  in  the  color-coded  measure-summary  table  has  a  flag  (K),  passing  the  mouse  over  it 
reveals  a  comment  (L)  with  a  description  of  the  option’s  risk  for  that  particular  measure. 
Finally,  passing  the  mouse  over  the  name  of  a  measure  brings  up  a  comment  (M)  from  the 
Measure  Comments  and  Weights  sheet  describing  the  measure  and  giving  its  weight  in  effec¬ 
tiveness  calculations.  In  addition,  clicking  on  the  Show/Fiide  Measure  Weights  checkbox 
(N)  displays  the  weights  of  all  of  the  measures  under  the  Detail  buttons.  The  measure 
weights  can  be  directly  modified  on  the  Summary  sheet,  and  the  sheet  can  be  recomputed  by 
clicking  on  the  Update  Measure  Weights  button  (O).  This  also  modifies  the  weights  of  the 
measures  on  the  Measure  Comments  and  Weights  sheet. 

Details  Sheet 

When  a  Details  button  is  clicked  on  the  Summary  sheet  for  a  particular  measure  (A  in  Figure 
4.2),  all  of  the  submeasures  (B)  for  that  particular  measure  are  displayed  on  the  Details  sheet, 
along  with  all  of  the  parameters  used  in  the  scoring  functions  (C)  associated  with  the  sub¬ 
measures. 

The  color  of  each  cell  in  the  table  will  depend  on  the  scoring  method  selected.  The 
default  scoring  method  was  used  to  generate  the  Summary  sheet,  although  this  can  also  be 
changed  using  the  Scoring  Method  dropdown  menu  on  the  Details  sheet  (D).  The  two-row 
color  code  at  the  bottom  of  the  table  (E)  explains  the  coloring  scheme  used  in  the  table  as 
well  as  the  one  used  to  color  the  results  of  the  summary  calculation  on  the  right-hand  side  of 
the  table.  The  latter  coloring  scheme  is  identical  to  the  one  used  in  the  measure-summary 
table  on  the  Summary  sheet  for  the  same  scoring  method. 
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Figure  4.2 
Details  Sheet 
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If  the  Goals  scoring  method  is  selected,  cells  will  be  green  or  red  depending  on 
whether  or  not  the  raw  value  for  the  submeasure  shown  reaches  the  goal.  The  rightmost  col¬ 
umn  (F)  shows  the  weighted  percentage  of  goals  achieved  over  all  of  the  submeasures  for  a 
given  measure,  and  the  color  of  the  cell  in  the  rightmost  column  is  the  same  as  that  of  the 
corresponding  cell  on  the  Summary  sheet. 

If  the  Thresholds,  Weak  Thresholds,  or  Weakest  Link  scoring  method  is  selected, 
cells  in  the  table  will  be  colored  red,  yellow,  or  green,  depending  on  whether  the  raw  value 
does  not  reach  the  threshold  value,  reaches  the  threshold  value  but  does  not  reach  the  goal 
value,  or  reaches  the  goal  value,  respectively.  The  color  of  the  cell  corresponds  to  the  measure 
score,  the  maximum  score  being  1  (corresponding  to  green)  and  the  minimum  score  being  0 
(corresponding  to  red). 

For  the  Rankings  scoring  method,  each  submeasure  is  ranked  individually,  with  the 
colors  of  the  cells  (from  light  blue  to  dark  blue)  based  on  how  well  the  investment  options 
rank  relative  to  one  another.  The  rightmost  column  shows  a  weighted-average  ranking  of  all 
of  the  submeasures,  with  the  color  of  the  cell  identical  to  the  color  of  the  corresponding  cell 
in  the  Summary  sheet  for  that  investment  option  and  measure. 

The  raw  values  and  the  scoring-function  parameters  for  each  submeasure  (except  the 
submeasure  names)  may  be  edited  in  the  Details  sheet,  even  though  it  is  nominally  an  output 
sheet.  After  the  data  and  parameters  have  been  changed,  clicking  on  the  Modify  Data  Entries 
button  (G)  will  update  the  Data  Entry  table  with  the  new  raw  values  and  will  also  recreate 
the  Summary  and  Details  sheets  with  the  new  information.  In  addition,  to  view  the  details 
of  another  measure,  the  user  can  replace  the  name  of  the  current  measure  with  that  of  the 
desired  measure  and  click  on  the  Get  Details  button  (H). 
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Scatter  Plot  Sheet 

In  the  Scatter  Plot  sheet,  the  user  can  choose  a  cost  metric  (A  in  Figure  4.3)  and  up  to  two 
evaluation  metrics  (B,C)  to  be  plotted.  The  cost  metrics  may  be  the  RDT&E,  deployment, 
or  total  investment  over  the  entire  time  period  under  consideration  or  one  of  the  user- 
defined  cost  metrics  on  the  Summary  sheet.  The  evaluation  metrics  may  be  submeasures 
from  the  Data  Entry  sheet  or  an  effectiveness  score  using  any  of  the  scoring  methods.  All  in¬ 
vestment  options  will  be  arrayed  on  a  scatter  plot  with  cost  as  the  x-axis  (D)  and  the  evalua¬ 
tion  metrics  presented  on  one  or  two  y-axes  (E,F).  Circular  points  (G)  indicate  the  perfor¬ 
mance  of  investment  options  relative  to  the  first  evaluation  metric,  and  square  points  (FI) 
indicate  the  second  evaluation  metric.  Passing  the  mouse  over  a  point  on  the  graph  lets  the 
user  know  which  investment  option  is  represented  by  that  point.  This  output  sheet  is  par¬ 
ticularly  useful  for  helping  to  determine  which  programs  should  be  dropped  because  of  re¬ 
duced  funding.  Starting  with  a  baseline  investment  option  (developing  a  particular  set  of 
ESGs),  one  can  produce  investment  options  corresponding  to  all  subsets  of  the  set  of  ESGs 
associated  with  the  baseline.  Then,  after  results  are  generated  from  the  capabilities  model,  the 
scatter  plot  can  highlight  (for  a  particular  evaluation  metric)  which  programs  should  be  re¬ 
tained,  based  on  the  size  of  the  funding  reduction.  One  simply  locates  the  collection  of  ESGs 
with  the  highest  value  of  the  evaluation  metric  to  the  left  of  the  new  funding  limit.  Changing 
the  evaluation  metric  or  any  of  the  investment  options  will  automatically  regenerate  the  scat¬ 
ter  plot,  as  will  clicking  on  the  Generate  Scatter  Plot  button  (not  pictured),  which  should  be 
used  when  the  raw  submeasure  values  or  the  parameters  on  the  Data  Entry  sheet  are  changed 
(the  scatter  plot  pulls  values  from  the  Data  Entry  sheet  only  when  the  chart  is  generated). 

Figure  4.3 

Scatter  Plot  Sheet  Showing  Two  Evaluation  Metrics  Versus  Cost 
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Spider  Charts  Sheet 

The  Spider  Charts  sheet2  allows  the  user  to  select  a  measure  (A  in  Figure  4.4)  and  up  to  four 
investment  options  to  be  included  in  the  spider  chart  (B,C).  Two  of  the  investment  options 
may  be  replaced  by  the  goal  and  threshold  values  for  the  submeasures  in  the  chart.  All  of  the 
submeasures  (of  non-zero  weight)  of  the  selected  measure  are  shown  as  arms  of  the  spider 
chart  (D).  The  values  in  the  chart  are  all  scaled  relative  to  the  investment  option  (or  goal  or 
threshold  value)  selected  as  Chart  1 .  In  addition,  the  raw  values  of  submeasures  for  which 
small  values  are  desirable  are  inverted  so  that  the  better  the  value,  the  further  outward  the 
corresponding  point  is  on  the  spider  chart.  There  are  no  values  on  the  axes  of  the  spider 
chart;  the  purpose  of  the  chart  is  to  show  the  relative  capabilities  of  each  investment  option 
for  a  particular  measure.  Changing  the  measure  to  be  charted  or  any  of  the  investment  op¬ 
tions  will  automatically  regenerate  the  spider  chart,  as  will  clicking  on  the  Generate  Spider 
Chart  button  (E),  which  should  be  used  when  the  raw  values  or  parameters  on  the  Data  En¬ 
try  sheet  are  changed  (the  spider  chart  pulls  values  from  the  Data  Entry  sheet  only  when  the 
chart  is  generated). 

Figure  4.4 

Spider  Charts  (Radar  Charts)  Sheet 
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2  Spider  charts  are  called  “radar  charts”  in  some  application  areas. 
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Multi-Measure  Spider  Charts  Sheet 

Whereas  the  Spider  Charts  sheet  provides  a  visual  depiction  of  a  given  measure’s  submea¬ 
sures,  the  Multi-Measure  Spider  Charts  sheet  (shown  in  Figure  4.5)  provides  a  visual  depic¬ 
tion  across  different  measures.  For  this  sheet,  the  arms  of  the  spider  chart  (A)  represent  dif¬ 
ferent  measures  (B).  For  each  investment  option  displayed  (C,D),  the  values  along  each  arm 
correspond  to  the  average  rank  for  the  Rankings  scoring  method  or  the  measure  score  for  the 
other  scoring  methods.  With  the  Thresholds  scoring  method,  if  any  investment  option  fails 
for  a  submeasure,  the  value  for  the  corresponding  measure  falls  inside  a  red  polygon  repre¬ 
senting  failed  measures.  The  user  can  select  the  scoring  method,  using  the  Scoring  Method 
dropdown  menu  (E),  and  up  to  four  investment  options.  Changing  the  measure  to  be 
charted  or  any  of  the  investment  options  will  automatically  regenerate  the  spider  chart,  as 
will  clicking  on  the  Generate  Spider  Chart  button  (F),  which  should  be  used  when  the  raw 
values  or  parameters  on  the  Data  Entry  sheet  are  changed  (the  spider  chart  pulls  from  the 
Data  Entry  sheet  only  when  the  chart  is  generated). 

Figure  4.5 

Multi-Measure  Spider  Charts  Sheet 
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Selected  Details  Sheet 

The  Selected  Details  sheet  (shown  in  Figure  4.6)  allows  the  production  of  a  sheet  similar  to 
the  Details  sheet,  except  that  the  submeasures  in  the  columns  are  specially  chosen  for  the 
purposes  of  the  particular  discussion  in  which  PAT-MD  is  being  used,  and  they  may  come 
from  different  measures.  This  allows  for  a  more  customized  display.  That  is,  whereas  the  De¬ 
tails  sheet  provides  a  zoom  of  a  particular  measure  (the  basis  for  a  particular  column  on  the 
Summary  sheet),  this  sheet  allows  the  user  to  tailor  a  display  with  details  selected  from  several 
measures.  As  with  the  Details  sheet,  all  scoring  methods  (and  the  associated  coloring 
schemes)  may  be  displayed  on  the  sheet  by  using  the  Scoring  Method  dropdown  menu  (A). 
To  display  a  particular  submeasure  in  a  column,  the  dropdown  menu  (B)  lists  all  of  the  sub¬ 
measures,  using  the  format  measure: :submeasure. 

Figure  4.6 

Selected  Details  Sheet 
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ESG  Table  Sheet 


The  ESG  Table  sheet  (sketched  in  Figure  4.7)  contains  a  collection  of  tables  and  charts  asso¬ 
ciated  with  the  cost  data  for  each  investment  option.  Seven  tables  provide  the  content  for  six 
charts  of  cost  data,  which  are  located  to  the  right  of  the  tables.  For  each  investment  option, 
the  tables  give,  in  order,  ESGs  available  (by  year),  ESG  availability  date  (by  ESG),  cumula¬ 
tive  RDT&E  cost  (by  year),  annual  RDT&E  cost,  and  a  breakdown  of  expenditures  by  the 
categories  defined  on  the  ESG  Data  sheet  (by  year).  Only  the  ESG  availability-date  table 
does  not  have  an  associated  chart.  The  charts  on  this  sheet  are  regenerated  each  time  the 
Summary  sheet  is  generated  or  when  the  Cost  Detail  button  over  any  of  the  cost-data  col¬ 
umns  on  the  Summary  sheet  is  clicked.  Note  that  the  charts  on  this  sheet  are  not  automati¬ 
cally  changed  when  the  Cost  Data  or  ESG  Data  input  sheet  is  modified. 


Rankings  Table  Sheet 

The  Rankings  Table  sheet  (shown  in  Figure  4.8)  ranks  the  investment  options  (E)  by 
effectiveness  (or  cost-effectiveness,  depending  on  whether  the  Rank  on  Cost-Effectiveness 
checkbox  (A)  is  checked)  for  each  perspective  (B)  defined  on  the  Perspective  Cases  sheet.  The 
scoring  method  used  to  compute  effectiveness  can  be  selected  by  the  user  from  the  Scoring 
Method  dropdown  menu  (C),  as  can  the  cost  metric  to  use  when  computing  cost- 

Figure  4.7 
ESG  Table  Sheet 
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Figure  4.8 

Rankings  Table  Sheet 


effectiveness  (if  applicable)  (D).  In  each  cell,  the  rank  is  in  large  bold  text  (F),  with  the  rela¬ 
tive  (cost-)  effectiveness  in  parentheses  under  the  ranking  (G).  The  investment  option  with 
the  highest  absolute  (cost-)  effectiveness  for  each  perspective  is  given  a  relative  (cost-)  effec¬ 
tiveness  of  1 ,  and  all  other  (cost-)  effectiveness  values  are  measured  against  it. 


IOC  to  Cost  Output  Sheet 

The  IOC  to  Cost  Output  sheet  (sketched  in  Figure  4.9)  contains  the  output  produced  when 
the  Generate  Cost  Data  button  is  clicked.  The  output  tables  are  identical  in  format  to  those 
used  as  inputs  in  the  Cost  Data  sheet.  The  results  can  be  copied  and  pasted  into  the  RDT &E 
columns  of  the  Cost  Data  sheet  as  the  cost  data  for  the  investment  cases  in  PAT-MD  or  can 
be  exported  to  CAMMD. 
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Figure  4.9 

IOC  to  Cost  Output  Sheet 
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5.  Details  of  the  Methodology 


Chapter  2  introduced  methodological  issues  in  PAT-MD,  using  a  very  simple  example.  This 
chapter  reviews  those  issues  and  provides  additional  generalizations,  details,  equations,  and 
subtleties.  It  first  presents  definitions,  then  equations  are  given  for  the  five  different  scoring 
and  aggregation  methods  available  in  PAT-MD.  Finally,  worked-out  examples  are  presented 
to  illustrate  how  effectiveness  and  cost-effectiveness  can  be  calculated  for  marginal  and 
chunky  marginal  analysis. 


Basic  Concepts  and  Definitions 

Figure  5.1  shows  schematically  how  PAT-MD  operates  analytically.1  For  each  investment 
option,  PAT-MD  takes  a  series  of  inputs  (the  shaded  items  in  Figure  5.1)  and  generates 
characterizations  of  how  well  the  option  performs  on  different  criteria,  including  cost- 
effectiveness.  The  concepts  and  terminology  of  Figure  5.1  are  explained  below. 

Investment  Options 

Alternative  investment  options  are  alternative  programs  of  investment  over  time.  They  are 
expressed  as  expenditures  (projected  costs)  for  each  year  treated  and  for  each  budget  category 
of  interest  in  the  analysis.  These  options  are  inputs  to  PAT-MD.  Most  budget  categories 
identify  systems  or  components  being  developed  or  deployed  (e.g.,  an  Aegis  radar  upgrade,  a 
midcourse-interceptor  missile,  or  a  kill  vehicle).  Other  budget  categories  might  cover  certain 
continuing  infrastructure  expenses.2 

Attributes  of  the  Investment  Options 

Each  investment  option  is  characterized  in  a  number  of  ways.  Some  of  these  characterizations 
are  input,  while  others  are  calculated  within  PAT-MD. 


1  Figure  5.1  is  not  precisely  correct  for  use  with  the  Rankings  method,  as  discussed  in  the  next  section. 

2  Analytically,  it  would  be  preferable  if  alternative  investment  options  were  fairly  narrow,  e.g.,  “develop  system  X”  versus 
“develop  system  Y.”  It  would  then  be  clear  that  in  characterizing  capabilities  or  risks,  we  were  referring  to  system  X  or  sys¬ 
tem  Y.  To  date,  however,  the  investment  options  for  our  MDA  analysis  have  been  total-program  options,  usually  described 
as  a  baseline  plus  or  minus  some  particular  systems  or  activities.  Such  a  total  program  must  be  characterized  along  a  number 
of  dimensions — hence,  the  multiple  measures  and  submeasures  discussed  below. 
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Figure  5.1 

Generation  of  Scores,  Effectiveness,  and  Cost-Effectiveness  in  PAT-MD 
(shaded  items  are  inputs) 
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Costs.  Each  option  may  have  a  number  of  types  of  cost,  such  as  one-year,  six-year,  or 
20-year  costs,  with  the  numbers  expressed  either  in  current  dollars  or  in  dollars  corrected  for 
inflation.  It  is  also  possible  to  use  “discounted  dollars.”  Although  that  information  is  seldom 
available  to  PAT-MD  as  an  input,  PAT-MD  can  transform  the  various  types  of  input  costs 
into  discounted  versions.3 

Measures  and  Submeasures.  The  options  are  characterized  by  a  number  of  criteria 
called  measures,  each  of  which  has  one  or  more  submeasures.  The  measures  and  submeasures 
typically  relate  to  types  of  capabilities  or  risk.4  They  are  akin  to  what  some  call  metrics. 


3  We  refer  here  to  making  “present-value  calculations,”  in  which  current  expenditures  are  treated  as  effectively  more  costly 
because  they  use  funds  that  could  otherwise  be  invested  in  other  ways  (i.e.,  agreeing  to  pay  someone  $1,000  in  ten  years  is 
much  less  painful  than  paying  him  $1,000  now,  because,  in  principle,  one  could  invest  the  $1,000  for  ten  years,  pay  the 
debt  at  that  point,  and  keep  the  returns).  The  adjustment  uses  compounding  with  the  so-called  discount  rate.  The  eco¬ 
nomic  calculations  are  well  understood  (Hitch  and  McKean,  1960),  although  economists  continue  to  differ  about  what 
discount  rate  should  be  assumed,  especially  for  government  expenditures.  For  tutorial  information  on  present-value  calcula¬ 
tions  in  defense  acquisition,  see  Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics,  n.d. 

4  Other  possibilities  exist.  If,  for  example,  space-based  defenses  were  somehow  more  psychologically  impressive  and  there¬ 
fore  more  valuable  for  deterrence  or  coercive  diplomacy,  one  measure  could  be  “psychological  value.”  During  the  Cold  War, 
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Submeasure  information  is  input  to  PAT-MD.  As  is  the  case  throughout  mathematics  and 
computer  science,  the  same  label  (e.g.,  “a  missile’s  single-shot  kill  probability”)  is  used  to  re¬ 
fer  to  both  the  abstract  concept  that  a  measure  or  submeasure  represents  and  the  value  of 
that  measure. 

Raw  Values  of  Submeasures.  A  submeasure’s  raw  value  is  data  input  to  PAT-MD. 
These  data  might  come  from  system  specifications  (with  the  assumption  that  the  specifica¬ 
tions  will  be  met),  from  test  data,  from  models,  or  from  expert  judgment.  The  raw  values 
may  be  expressed  in  different  units  and  on  different  scales,  and  a  larger  raw  value  may  be 
good  or  bad,  depending  on  the  submeasure. 

Submeasures  do  not  necessarily  convey  directly  any  sense  of  sufficiency.  For  example, 
if  an  option  generates  a  defense  system  that  could  be  saturated  by  12  attackers  (12  would  be 
the  raw  value  of  a  submeasure),  this  does  not  indicate  whether  12  is  good  or  bad,  sufficient 
or  insufficient. 

Goals  and  Thresholds.  Most  of  the  methods  used  in  PAT-MD  involve  goals  or  both 
goals  and  thresholds  (again,  the  Rankings  method  is  the  exception).  Inputs  to  PAT-MD  then 
include,  for  each  submeasure,  a  raw  value  corresponding  to  the  goal  and  possibly  a  raw  value 
corresponding  to  a  threshold  (a  minimum  level  for  perceived  utility). 

Submeasure  Scores.  PAT-MD  calculates  the  score  of  a  submeasure  from  its  raw 
value,  other  inputs  regarding  goals  and  thresholds,  and  the  raw  values  of  the  submeasure  for 
other  options.  In  PAT-MD,  for  all  methods  except  the  Rankings  method,  the  score  is  be¬ 
tween  0  and  1,  with  1  being  good. 

Measure  Scores.  The  score  of  a  measure  is  calculated  from  the  scores  of  its  submea¬ 
sures;  that  is,  a  measure’s  score  is  an  aggregation  of  its  submeasure  scores.5  Except  under  the 
Rankings  method,  measure-level  scores  (or,  simply,  measure  scores)  are  also  between  0  and  1 . 

Composite  Score  (Effectiveness).  Once  PAT-MD  has  calculated  scores  for  the  mea¬ 
sures  characterizing  the  investment  options,  it  can  generate  composite  scores  of  effectiveness, 
or  simply  effectiveness.  And  it  can  use  the  highest  such  score  as  a  base  and  compute  the  rela¬ 
tive  effectiveness  of  the  investment  options. 

Relative  Cost-Effectiveness 

Cost-effectiveness  is  calculated  using  the  ratio  of  effectiveness  and  cost.  With  PAT-MD,  an 
analyst  can  use  any  of  the  input  cost  categories  (e.g.,  six-year  costs  in  real  dollars)  as  the  basis 
for  the  cost-effectiveness  calculation.  PAT-MD  calculates  an  intermediate  cost-effectiveness 
for  each  option  (not  shown  in  Figure  5.1)  and  then  uses  the  largest  value  as  the  base,  to 
which  it  compares  all  the  options’  cost-effectiveness.  The  result,  then,  is  the  relative  cost- 
effectiveness  for  each  option. 


the  Soviets  probably  earned  far  more  perception  points  for  their  large-throw-weight  ICBMs  than  they  deserved,  from  a 
technical  standpoint.  So  also,  the  American  Strategic  Defense  Initiative  (SDI)  had  a  greater  impact  on  Soviet  thinking  than 
was  merited  by  its  technical  success  at  the  time. 

5  A  more  rigorous  term  is  “abstraction.”  That  is,  a  measure  is  an  abstraction  from  its  subordinate  submeasures.  It  abstracts 
from,  or  captures  the  relevant  essence  of,  the  submeasures.  It  may  be  a  simple  average,  or  it  may  be  a  more  context-sensitive 
projection.  Historically,  “aggregation”  meant  the  result  of  collecting,  as  one  might  aggregate  the  strengths  of  nine  battalions 
to  estimate  the  strength  of  a  division.  Perhaps  unfortunately,  “aggregation”  is  coming  to  have  a  more  general  meaning  akin 
to  “abstraction.”  In  software,  for  example,  aggregation  is  sometimes  seen  as  a  composition  technique  for  creating  a  new 
object,  in  part,  by  using  component  objects.  For  a  more  careful  discussion,  see  Zeigler,  Zeigler,  and  Praehofer,  2000. 
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Methods  and  Functions 

The  progression  summarized  in  Figure  5.1  requires  numerous  functions  (F1,F2,..,F5),  as  indi¬ 
cated.  These  accomplish  the  following  transformations: 

•  Submeasure  raw  values  to  submeasure  scores 

•  Submeasure  scores  to  measure-level  scores 

•  Costs  of  various  types  to  a  single  cost  used  in  cost-effectiveness  calculations  6 

•  Measure-level  scores  to  a  composite  score  (effectiveness) 

•  Effectiveness  scores  and  cost  to  cost-effectiveness 

The  different  methods  in  PAT-MD  to  which  we  have  alluded  use  different  functions,  as  dis¬ 
cussed  below. 

Table  5.1  summarizes  the  terms  used  in  PAT-MD  and  their  meanings  and  includes 
an  example  of  each. 

Table  5.1 

PAT-MD  Terminology 


Term 

Meaning  in  PAT-MD 

Example 

Abstraction 

A  generalization  derived  from  specific  more- 
detailed  or  more-concrete  cases,  perhaps  with 
a  particular  context  of  use  in  mind.3  Measures 
are  abstractions  of  submeasures. 

Engagement  effectiveness  is  an  abstraction 
derived  from  radar,  missile,  and  kill-vehicle 
effectiveness. 

Aggregation 

Cost-effectiveness 

Abstraction. 

The  ratio  of  an  effectiveness  score  to  a  measure 
of  cost. 

Ten-year  cost  is  a  simple  aggregation  of  one- 
year  costs.  Overall  risk  of  an  option  might  be 
a  more-complex  aggregation  of  technical, 
strategic,  and  political  risks. 

Effectiveness 

A  composite  score  formed  by  abstracting  from 
the  scores  of  one  or  more  measures. 

A  given  investment  option  might  have  a  score 
of  7,  reflecting  high  technical  effectiveness 
in  some,  but  not  all,  missions  and  situations. 

Measure 

A  way  of  evaluating  something;  a  dimension 
of  an  assessment.  An  investment  option  is 
assessed  by  different  measures.  Measure  is 
similar  to  metric. 

The  size  of  attack  that  saturates  a  defensive 
system. 

Method 

A  procedure  used  to  map  raw  scores  into 
scores  or  to  calculate  higher-level  scores 
from  lower-level  scores. 

A  measure's  score  might  be  the  simple  aver¬ 
age  of  its  submeasure  scores. 

Raw  value  of  a 

An  unsealed  value  of  a  submeasure,  which 

The  size  of  attack  that  saturates  a  defensive 

submeasure 

Relative  effective¬ 
ness 

Relative  cost- 
effectiveness 

may  not  itself  convey  a  sense  of  goodness 
or  utility. 

The  ratio  of  an  effectiveness  score  to  the 
highest  such  score  among  options. 

The  ratio  of  an  option's  cost-effectiveness  to 
that  of  the  option  with  the  highest  cost- 
effectiveness. 

system  in  the  absence  of  offensive  counter¬ 
measures  might  be  12.  Is  that  good  or  bad? 

Score 

A  value  derived  from  raw  values  and  goals  to 
convey  a  sense  of  goodness. 

A  score  of  0.6  might  be  calculated  as  the  ratio 
of  a  raw  score  of  12  and  a  goal  value  of  20, 
scaled  to  be  between  0  and  1. 

Submeasure 

One  of  the  factors  determining  a  parent 
measure. 

The  size  of  attack  that  saturates  a  defensive 
system  in  a  particular  case  (e.g.,  with  at¬ 
tacker  countermeasures). 

a  The  average  depth  in  the  deep-water  area  of  a  swimming  pool  might  be  an  example.  It  is  not  the  pool's  overall 
average  depth  but,  rather,  an  average  relevant  to  safety  calculations  concerning  diving  boards.  In  physics,  "projec¬ 
tion  operators"  generate  such  pointed  abstractions. 


6  Currently,  this  is  implemented  in  PAT-MD  as  a  selection  rather  than  an  aggregation.  That  is,  a  user  who  enters  costs  in 
various  forms  can  select  which  form  to  use  in  cost-effectiveness  calculations. 
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Alternative  Methods  Used  in  PAT-MD 

The  Need  for  Alternative  Methods 

As  discussed  above,  each  of  the  steps  in  Figure  5.1  requires  specifying  the  appropriate 
mathematical  function  (e.g.,  for  mapping  raw  values  into  scores  or  for  calculating  cost- 
effectiveness).  In  a  given  application  of  PAT-MD,  the  analyst  chooses  a  scoring  method. 
Each  such  method  must  specify  the  five  functions  indicated  in  Figure  5.1  (F1,...,F5).  No 
single  “right”  method  exists,  even  though  the  bulk  of  work  in  mathematical  decision  analy¬ 
sis  uses  a  single  method  based  on  linear  weighted  sums  contributing  to  a  single  measure  of 
“utility.”  That  standard  method  is  taught  in  many  schools  and  is  embedded  without  com¬ 
ment  in  much  decision-support  software.7  It  is  often  quite  useful,  just  as  assuming  that  the 
probabilities  of  various  factors  affecting  a  problem  are  independent  is  often  quite  useful. 

Nonetheless,  in  capabilities-based  planning  (and  in  much  of  systems  analysis  and 
policy  analysis  more  generally),  the  standard  approach  is  insufficient  for  several  reasons: 

•  Decisionmakers  need  to  know  separately  about  some  of  the  “apples  and  oranges,”  and 
total  aggregation  into  a  single  utility  suppresses  too  much  information  and  depends 
too  heavily  on  underlying  assumptions  and  preferences  that  are  the  decisionmaker’s 
province  rather  than  the  analyst’s.8 

•  Similarly,  decisionmakers  need  to  know  how  a  given  option  addresses  each  of  their 
separate  high-level  objectives.9 

•  Aggregation  rules  sometimes  need  to  be  nonlinear  because  of  system  effects. 

To  elaborate  for  the  MDA  context,  it  is  important  that  decisionmakers  be  able  to  see 
information  about  BMDS  capabilities  against  small  and  large  attacks  and  attacks  with  and 
without  countermeasures  separately.  No  single  measure  of  BMDS  performance  is  adequate. 
Similarly,  MDA  has  different  missions  and  different  objectives  for  each  mission.  At  a  more 
technical  level,  MDA’s  capabilities-based  planning  is  concerned  with  systems  and  with  sys¬ 
tems  of  systems  (e.g.,  the  overall  BMDS).  The  result  is  a  need  to  highlight  the  importance  of 
individual  critical  components  and  to  be  concerned  with  interdependencies  rather  than  sup¬ 
pressing  them.  This  implies  a  need  to  allow  for  nonlinearities  in  the  mathematics  used 
(Davis,  2002). 

Suppose,  for  example,  that  a  system  has  three  critical  components:  a  radar,  an  inter¬ 
ceptor,  and  a  kill  vehicle.  Since  they  are  all  critical,  a  valuation  scheme  might  weight  them 
equally  in  a  linear  sum:  system  quality  =  (radar  quality)  +  (interceptor  quality)  +  (kill-vehicle 
quality) . 


7  The  classic  introductory  book  on  decision  analysis  (Raiffa,  1968)  is  quite  readable.  A  later  text  (Keeney  and  Raiffa,  1976) 
treats  multi-objective  decision  analysis.  Although  it  emphasizes  combining  into  a  single  utility,  Keeney  and  Raiffa  (1976) 
also  discusses  alternatives  to  simple  linear  weighted  sums. 

8  This  consideration  has  led  to  the  current  emphasis  on  scorecards  in  policy  analysis.  Such  analytic  summaries  emerged  in¬ 
dependently  in  a  number  of  places,  but  an  early  in-depth  application  with  considerable  methodological  discussion  was  the 
analysis  of  a  Netherlands  water-management  problem  (Goeller,  1983).  PAT-MD  is  a  modern  manifestation. 

9  A  form  of  multi-objective  decision  analysis  that  organizes  around  the  user’s  objectives  is  value-focused  thinking  (Keeney, 
1992).  It  is  currently  popular  for  many  applications,  including  work  in  military  organizations  (Parnell,  Burk,  Westphal,  et 
al„  2004). 
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Or  suppose  that  improving  the  radar  is  very  inexpensive  because  it  is  a  simple  modi¬ 
fication  of  an  existing  radar,  whereas  improving  the  kill  vehicle  is  extremely  expensive.  If  the 
above  equation  were  correct,  then,  even  with  weighting  factors,  a  seemingly  cost-effective 
investment  strategy  would  be  to  put  all  of  the  funds  into  the  radar — the  mathematical  impli¬ 
cation  of  the  assumed  linearity.  But  that  investment  would  be  absurd. 

To  reflect  such  common  system-related  issues,  something  other  than  simple  linear 
sums  is  needed.  A  number  of  possibilities  exist,  ranging  from  using  a  multiplicative  relation¬ 
ship  instead  of  an  additive  one  to  methods  that  involve  enforcing  threshold  requirements  for 
each  of  the  critical  components.  In  PAT-MD  terminology,  this  corresponds  to  enforcing 
threshold  requirements  for  each  of  the  submeasures  that  characterize  an  investment  option. 

Therefore,  we  have  built  five  distinct  methods  into  PAT-MD.  This  has  the  great  ad¬ 
vantage  of  increasing  flexibility  but  the  disadvantage  of  causing  potential  confusion  and  un¬ 
dercutting  the  goal  of  providing  relatively  simple,  logical,  and  intuitive  results  for  decision¬ 
makers.  Thus,  in  what  follows,  we  describe  the  five  methods;  the  analyst,  however,  should 
decide  which  method  or  methods  to  use  and  should  then  present  only  those.  The  five 
methods  are  (1)  goal-based  (Goals,  for  short),  (2)  goal-based  with  weak  thresholds  (Weak 
Thresholds),  (3)  goal-based  with  thresholds  (Thresholds),  (4)  goal-based  with  weakest  link 
(Weakest  Link),  and  (5)  rankings-based  (Rankings). 

These  are  described  separately  in  what  follows  and  are  then  summarized  in  a  table. 
For  each  method,  the  description  specifies  how  scores  are  generated  from  the  raw  values  of 
submeasures  and  how  higher-level  scores  and  effectiveness  are  calculated  by  aggregation. 

We  use  a  common  notation,  as  defined  in  Table  5.2. 


Goals  Method 

The  Goals  method  is  the  simplest  method  to  describe  and  is  often  appropriate.  Every  mea¬ 
sure  consists  of  a  collection  of  submeasures,  each  of  which  has  a  goal  value  that  each  invest¬ 
ment  option  is  trying  to  achieve  for  it.  For  example,  a  kill  device  might  have  two  kill  mecha¬ 
nisms,  A  and  B,  either  of  which  would  be  sufficient.  In  designing  the  device,  an  engineer 
would  have  a  goal  for  the  capability  of  each  mechanism. 

Submeasure  Scores  for  the  Goals  Method.  The  score  of  a  submeasure  is  0  or  Gjj, 
depending  on  whether  the  submeasure’s  raw  value  has  reached  the  submeasure’s  goal.  That 
is,  for  any  investment  option  i  and  any  measure  j,  if  the  scale  is  increasing  so  that  goals  corre¬ 
spond  to  high  values,  then  the  £th  submeasure’s  score  is  given  by 


=  0  if  ViJJt  <  V% 
=  G, ,  if  V. , ,,  >  V, 


' j.k 


i,j,k 


j,k 


(5.1) 


If  goodness  increases  with  decreasing  raw  values,  then  the  equations  change  accordingly. 
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Table  5.2 

Notation  for  Defining  Scoring  Methods 


Symbol 


Meaning 


Parameters 

i,j,k 

m 

ni 

2, 

E, 

Parameter  values  used  in  c 

Kg* . 

VT  } 

’  vJ-»  J 

KK . 

VG  i 
’  vj-«  J 

Ra, . 

{g,„g;2,„ 

•G,.} 

R 

{CpC2,...,Cm} 


Indexes  for  investment  option,  measure,  and  sub¬ 
measure,  respectively 

Number  of  measures 

Number  of  submeasures  of  measure  j 

Raw  values  of  the  submeasures  for  investment  option  / 
and  measure  j 

Scores  of  the  n;  submeasures  (submeasure  scores)  for 
investment  option  /  and  measure  j 

Scores  of  the  m  measures  (measure  scores)  for 
investment  option  i 

Composite  score  (effectiveness)  for  investment  option  / 


Threshold  raw  values  for  each  submeasure  of  measure  j 

Goal  raw  values  corresponding  to  each  submeasure  of 
measure  j 

Submeasure  scores  when  submeasure  raw  values  have 
reached  thresholds,  under  measure  j 

Scores  of  submeasures  that  have  reached  their  goals 
under  measure  j 

Weights  of  the  submeasures  in  computing  measure 
scores 

Weights  of  the  measures  in  computing  effectiveness 


Source 

n/a 

n/a 

n/a 

Input  ( Data  Entry  sheet) 
Calculated 

Calculated 

Calculated 

Input  ( Data  Entry  sheet) 

Input  ( Data  Entry  sheet) 

Input  ( Data  Entry  sheet) 

Input  ( Data  Entry  sheet) 

Input  ( Data  Entry  sheet) 

Input  ( Measure  Comments 
and  Weights  sheet,  or 
Summary  sheet) 


Aggregation  to  Measure  Scores  for  the  Goals  Method.  The  function  used  to  calcu¬ 
late  the  score  of  a  measure  is  simply  a  weighted  sum  of  the  submeasure  scores. 


I»0Am 


M.  .  = 


—  *=i 


i,J  n- 


5RG 


j,k 


k= 1 


(5.2) 


If  all  submeasures  were  equally  weighted,  the  measure  score  would  be  the  fraction  of  the 
measure’s  submeasures  that  reached  their  goals.  The  resulting  measure  scores  are  between  0 
and  1  because  of  the  normalization  accomplished  by  the  denominator.  In  practice,  the  values 
of  Gj  k  will  all  be  equal  to  1,  with  the  Wjk  values  establishing  the  relative  weights  of  the  sub¬ 
measures,  but  for  the  sake  of  completeness,  we  include  Gjk  in  our  equations. 
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Aggregation  to  Composite  Score  (Effectiveness)  for  the  Goals  Method.  The  function 
used  to  set  the  composite  score  or  effectiveness  under  investment  option  i  is  also  a  weighted 
sum  over  measures: 


E,=—m -  (5-3) 

Ec. 


The  measure  and  effectiveness  scores  all  have  values  between  0  and  1 .  The  weights 
need  not  be  between  0  and  1,  since  PAT-MD  performs  normalizations,  but  it  is  arguably 
good  practice  to  enter  weights  in  that  range  that  add  up  to  1 .  This  requires  some  additional 
work  on  the  part  of  the  analyst  entering  data,  but  in  our  experience,  the  data  are  sometimes 
easier  to  understand  and  adjust. 


Weak  Thresholds  Method 

Submeasure  Scores  for  the  Weak  Thresholds  Method.  Under  this  method,  each 
submeasure  has  a  threshold  as  well  as  a  goal.  If  the  threshold  is  not  reached,  the  submeasure 
score  is  0.  If  the  submeasure  reaches  or  exceeds  its  goal,  the  submeasure  score  is  the  submea¬ 
sure  score  for  the  goal  value.  In  between,  the  score  is  determined  by  a  linear  relationship. 
That  is,  in  cases  where  goodness  increases  with  raw  value, 


i.j,k 


Jij,k 


T 

j,k 


Jj,k 


ifV<VJ 
ifV>V l 


■  T, ,  +  \*  Tjf  (Gjk  -T.k)  if  Vfk  <  V,  <  V) 


j>k 


VG  -VT 

'  J'k  V  j,k 


G 

j,k 


(5.4) 


If  goodness  decreases  with  raw  value,  then  the  equations  must  be  adjusted. 

Figure  5.2  shows  the  resulting  scoring  method. 

Aggregation  to  Measure  Score  for  the  Weak  Thresholds  Method.  The  score  of 
measure  j  is  again  given  by  a  linear  weighted  sum,  as  in  the  Goals  method,  but  the  submea¬ 
sure  scores  are  different: 


2XAm 


M 


k= 1 


i,J  n- 


XX. G 


j.k 


k=l 


(5.5) 
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Figure  5.2 

Submeasure  Score  Versus  Submeasure  Raw  Value  for  Goals  and  Thresholds  Methods 

Submeasure  score 


Aggregation  to  Composite  Score  (Effectiveness)  for  the  Weak  Thresholds  Method. 

The  composite  score  (effectiveness)  is  given  by  a  linear  weighted  sum,  as  previously,  but  the 
measure  scores  are  different: 


E.  =— -  (5.6) 

i  m  v  7 

Ec, 


Thresholds  Method 

Submeasure  Scores  for  the  Thresholds  Method.  Submeasure  scores  are  calculated 
with  the  same  function  as  that  used  for  the  Weak  Thresholds  method: 


stJJl  =  o 


if  V  <  V , 


j,k 


ifv>v ; 


j,k 


ij,k 


■  T, ,  +  V,J*  T]:k(GLk-TLk)  if  vj,  <  vu ,,  <  v; 


J'k 


VG  -  VT 

V  j.k  V  j,k 


G 

j,k 


(5.7) 
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Aggregation  to  Measure  Scores  for  the  Thresholds  Method.  The  difference  in  meth¬ 
ods  occurs  here.  If  any  submeasure  fails  to  reach  its  threshold,  the  measure  is  assigned  a  score 
of  0: 


M 


k=l 


if  all  submeasures  reach  their  thresholds 


j,k 


k= 1 


MtJ  =  0 


otherwise 


(5.8) 


Aggregation  to  Composite  Score  (Effectiveness)  for  the  Thresholds  Method.  Effec¬ 
tiveness  is  calculated  precisely  as  before,  but  the  measure  scores  are  different: 


-  (5.9) 

EC, 

i=l 


Weakest  Link  Method 

The  Weakest  Link  method  is  even  more  stringent  than  the  Thresholds  method  in  enforcing 
the  “requirements”  represented  by  thresholds. 

Submeasure  Scores  for  the  Weakest  Link  Method.  These  submeasure  scores  are  cal¬ 
culated  with  the  same  function  as  that  used  for  the  Thresholds  method: 


o 

II 

•4* 

tf  V  <  Vjtk 

i,j,k  Gj,k 

V,  ik~Tik 

if  V>  V?k 

(5.10) 

_  'T’  |  1>J>K  J->k 

i,j,k  ~1j,k  yG  _yT  W j,k 

~Tj,k)  if  Vjf  <  Vi  jk  <  V?k 

Aggregation  to  Measure  Scores  for  the  Weakest  Link  Method.  Under  the  Weakest 
Link  method,  the  measure  score  is  the  minimum  of  the  submeasure  scores.  If  any  submea¬ 
sure  fails  to  reach  its  threshold,  the  measure  score  will  be  0,  as  in  the  Thresholds  method,  but 
if  all  submeasures  reach  their  thresholds,  the  score  here  will  be  different  from  that  for  the 
Thresholds  method  and,  typically,  smaller: 

Mij  =  mm{Sijk}  (5.11) 

Aggregation  to  Composite  Score  (Effectiveness)  for  the  Weakest  Link  Method.  Ag¬ 
gregation  under  the  Weakest  Link  method  is  performed  by  taking  the  minimum  measure 
score  (which  is  identical  to  the  minimum  submeasure  score  over  all  submeasures).  Note  that 
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if  even  one  submeasure  in  any  of  the  measures  fails  to  reach  its  threshold,  effectiveness  will  be 
0.  Even  if  that  does  not  happen,  effectiveness  will  typically  be  smaller  with  this  method: 

Ei=rmn{Mij}  (5.12) 


Color  Coding  for  the  Goals  and  Thresholds  Methods 

For  all  the  non-rankings  methods  in  PAT-MD,  the  scores  of  the  investment  options  can  be 
mapped  into  the  colors  of  familiar  stoplight  charts,  where  red  is  worst  and  orange,  yellow, 
light  green,  and  dark  green  are  successively  better.  A  five-color  system  is  used  for  our  meas¬ 
ure-summary  table  because,  over  the  decades,  five  has  proven  to  be  a  comfortable  number 
that  makes  sufficient  distinctions  but  avoids  cognitive  overload.  The  mapping  for  the  meas¬ 
ure  scores  is  shown  in  Table  5.3.  For  the  Thresholds  and  Weakest  Fink  methods,  if  any 
submeasure  fails  to  reach  a  threshold,  the  cell  in  the  measure-summary  contains  an  F  in  the 
corner. 

Table  5.3 

Mapping  Measure  Scores  into  Colors 

Range  of  Scores  Color 

0  <  Score  <  0.2 
0.2  <  Score  <  0.4 
0.4  <  Score  <  0.6 
0.6  <  Score  <  0.8 
0.8  <  Score  <  1 


Yellow 
Light  green 


Dark  green 


Rankings  Method 

The  Rankings  method  does  not  use  goal  or  threshold  values.  Instead,  for  each  submeasure, 
the  investment  options  are  simply  ranked  from  best  to  worst,  without  regard  to  absolute  per¬ 
formance.  For  this  method,  we  refer  to  submeasure  and  measure  ranks  instead  of  submeasure 
and  measure  scores. 

Submeasure  Ranks.  Fet  Ri  ■  k  be  the  rank  of  investment  option  i  for  measure  ^  and 
submeasure  k.  We  define  it  as  one  more  than  the  number  of  investment  options  that  perform 
strictly  better  than  investment  option  i  on  submeasure  k  of  measure  j.  Thus,  if  two  invest¬ 
ment  options  have  the  same  raw  value,  they  will  have  the  same  rank. 

Aggregation  to  Measure  Ranks  for  the  Rankings  Method.  The  aggregation  from 
submeasure  rankings  to  measure  rankings  is  again  a  linear  weighted  sum.  We  calculate  the 

average  rank  R  .  within  measure  j: 


 M 


7=1 


(5.13) 
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The  value  Ri  .  is  used  to  set  the  color  (one  of  five  shades  of  blue)  in  the  measure-summary 
table,  depending  on  the  quintile  in  which  the  average  ranking  resides. 

Aggregation  to  Composite  Effectiveness  for  the  Rankings  Method.  This  aggregation 
function  is  best  explained  with  words  and  examples.  It  is  entirely  different  from  the  functions 
used  in  the  other  methods  in  that  each  measure  is  assigned  a  score  based  on  rankings. 

Let  us  assume  that  ^investment  options  are  under  consideration,  and  each  submea¬ 
sure  has  K(K—  1)12  points  to  distribute  among  those  options.  If  there  are  no  ties  for  the  val¬ 
ues  in  submeasure  j,  then  the  submeasure  score,  R' ;  k ,  is  given  by 


Kj,  =  K-R,.,k 


(5.14) 


If  there  are  ties,  the  points  that  would  have  gone  to  the  investment  options  in  those 
positions  are  combined  and  then  equally  distributed  among  the  options.  For  example,  with 
ten  investment  options,  the  second-  and  third-place  investment  options  would  receive  8  and 
7  points,  respectively.  If  two  investment  options  are  tied  for  second  place,  they  would  receive 
(7  +  8)/2  =  7.5  points  each.  This  scoring  method  is  used  instead  of  a  linear  transformation  of 
the  rankings  to  avoid  artificially  large  numbers  in  the  case  of  ties.  To  illustrate,  if  ten  invest¬ 
ment  options  had  the  same  value  for  a  submeasure,  they  would  all  tie  for  first.  If  the  ranking 
effectiveness  score  did  not  take  ties  into  account,  each  investment  option  would  receive  9 
points,  so  the  total  (unweighted)  contribution  to  the  final  score  (summed  across  investment 
options)  by  that  submeasure  would  be  90  points.  By  comparison,  for  a  submeasure  where 
every  investment  option  had  a  different  value,  the  total  (unweighted)  contribution  to  the 
final  score  by  that  submeasure  would  be  45  points  (9  +  8  +  7  +  6  +  5  +  4  +  3  +  2+  l).10 

As  in  the  previous  scoring  methods,  each  submeasure  Sjk  has  a  weight  Wj  k  in  the  ag¬ 
gregation  to  the  measure  score.  Similarly,  the  weights  C-  determine  the  relative  contribution 
of  each  measure  to  the  effectiveness  score  for  each  investment  option.  The  measure  score  for 
an  investment  option  i  for  measure  j  is 


tv . 


M,  ..  = 


k= 1 


ij  n : 


^(K-DWj, 


j,k 


k= 1 


(5.15) 


Because  the  maximum  possible  value  of  R'  ■  k  for  each  individual  submeasure  is 
K—  1,  the  denominator  in  Equation  5.15  scales  the  effectiveness  score  to  be  between  0  and  1. 
The  effectiveness  score  for  each  investment  option  over  all  measures,  Ep  is  given  by 


E  =  ^~ 


hj 


rn 

j= i 


(5.16) 


10  This  method  is  a  slight  modification  of  the  Borda  count  voting  method,  allowing  for  ties  (see  Ratliff,  2002).  TheBorda 
count  method  is  used  in  many  organizations  and  is  also  used  to  determine  the  Associated  Press  college  football  rankings. 
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Color  Coding  for  the  Rankings  Method 

Color  coding  on  the  measure-summary  table  for  the  Rankings  method  is  different  and  even 
uses  different  colors,  in  order  not  to  convey  an  impression  of  good  and  bad  as  the  stoplight 
charts  do  for  the  Goals  and  Thresholds  methods. 

The  weighted  average  rank  determines  the  colors  on  the  Summary  sheet  for  each 
measure.  As  shown  in  Table  5.4,  the  colors  go  from  light  blue  to  dark  blue,  denoting  the 
quintile  in  which  the  weighted  average  rank  of  the  investment  option  lies.  For  example,  if 
there  were  ten  investment  options,  an  investment  option  would  receive  the  lightest  blue  color 
if  its  average  rank  was  two  or  lower  and  would  receive  the  darkest  blue  color  if  its  weighted 
average  rank  was  eight  or  greater.  The  same  coloring  method  is  used  on  the  Details  sheet, 
where  the  rank  of  each  submeasure  determines  the  color  of  the  cell. 

Table  5.5  summarizes  the  five  methods  built  into  PAT-MD.  It  is  somewhat  cryptic, 
but  the  compact  comparison  may  be  helpful  to  readers. 

Table  5.4 

Color  Coding  for  the  Rankings  Method 

Meaning  Color 


(Average)  rank  in  first  quintile 
(Average)  rank  in  second  quintile 
(Average)  rank  in  third  quintile 
(Average)  rank  in  fourth  quintile 
(Average)  rank  in  fifth  quintile 


Table  5.5 

Summary  of  Methods 


Method 

Submeasure 

Scores 

Measure  Scores 

Composite  Score 
(Effectiveness) 

Coloring 

Method 

Comment 

Goals 

0  or  1,  depend¬ 
ing  on  whether 
goal  is  reached 

Linear  weighted  sum  of 
submeasure  scores 

Linear  weighted 
sum  of  mea¬ 
sure  scores 

As  in  Table  5.3 

Simple  and  common 
but  has  sharp  edges 

Thresholds 

As  in  Figure  5.2 

Zero  if  any  submeasure 
fails  to  reach  threshold; 
otherwise,  linear 
weighted  sum  of  sub¬ 
measure  scores 

Linear  weighted 
sum  of  mea¬ 
sure  scores 

As  in  Table  5.3 

May  be  appropriate  if 
all  submeasures  are 
critical  and  have  firm 
requirements 

Weak 

Thresholds 

As  in  Figure  5.2 

Linear  weighted  sum  of 
submeasure  scores 

Linear  weighted 
sum  of  mea¬ 
sure  scores 

As  in  Table  5.3 

May  be  appropriate  if 
not  all  submeasures 
are  critical 

Weakest 

Link 

As  in  Figure  5.2 

Minimum  of  submea¬ 
sure  scores 

Minimum  of 
measure  scores 

As  in  Table  5.3 

May  be  appropriate  if 
all  measures  and 
submeasures  are  criti¬ 
cal  and  have  firm 
requirements 

Rankings 

Modified  Borda 
count 

Weighted  average  of 
ranks  for  measure- 
summary  table,  linear 
weighted  sum  of  Borda 
scores  for  effectiveness 

Linear  weighted 
sum  of  mea¬ 
sure  scores 

As  in  Table  5.4 

May  be  appropriate  if 
one  wishes  to  avoid 
discussion  of  goals 
and  thresholds 
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Some  Mathematical  Observations 

All  of  the  scoring  and  aggregation  functions  used  in  PAT-MD  are  monotonic  nondecreasing. 
Thus,  improving  the  performance  of  an  investment  option  in  some  submeasure  cannot  de¬ 
crease  the  effectiveness  score.  The  aggregations  to  an  overall  effectiveness  score  have  the  same 
property.  More  to  the  point,  our  scoring  methods  do  not  lead  to  counterintuitive  conclu¬ 
sions  (except  in  degenerate  cases  of  no  significance).  We  do  not  require  that  the  functions  be 
suitable  for  optimization.11  This  said,  it  would  be  desirable  to  have  more  comprehensive  and 
structured  mechanisms  for  exploratory  analysis  with  PAT-MD,  and  this  may  be  addressed  in 
future  work.  In  the  meantime,  much  can  be  accomplished  with  the  alternative-perspectives 
mechanism  discussed  in  Chapters  3  and  4. 


Examples  of  Scoring  and  Aggregation  with  the  Different  Methods 


To  provide  examples  for  each  scoring  method,  we  will  look  at  two  measures,  M t  and  M2, 
each  consisting  of  three  submeasures.  All  submeasures  have  values  between  0  and  10,  inclu¬ 
sive,  with  2  being  the  threshold  value  and  7  being  the  goal  value  in  each  case.  For  both 
measures,  submeasure  Si  .  will  have  weights  twice  that  of  the  other  two  submeasures,  with  all 
submeasures  receiving  a  score  of  0  for  reaching  the  threshold  value.  By  setting  the  weights  of 
the  three  submeasures  to  0.25,  0.25,  and  0.5,  the  sum  of  the  weights  is  1,  so  there  is  no  need 
to  divide  the  sum  of  the  submeasure  scores  by  the  sum  of  the  weights,  and  we  will  assume 
the  weights  of  the  measures  are  equal.  We  will  consider  three  investment  options,  denoted  A, 
B,  and  C.  The  raw  values  for  each  option  for  each  submeasure  are  given  in  Table  5.6. 

Table  5.6 

Scoring  Methods  for  Three  Investment  Options 


Measure 

Ml 

Ml 

Ml 

M2 

M2 

M2 

Submeasure 

S1,1 

SI, 2 

SI, 3 

S2,1 

S2,2 

S2,3 

Submeasure  Weight 

0.25 

0.25 

0.5 

0.25 

0.25 

0.5 

Goal  Value 

7 

7 

7 

7 

7 

7 

Goal  Submeasure  Score 

1 

1 

1 

1 

1 

1 

Threshold  Value 

2 

2 

2 

2 

2 

2 

Threshold  Submeasure  Score 

0 

0 

0 

0 

0 

0 

Investment  Option  A 

10 

10 

5 

5 

5 

5 

Investment  Option  B 

7 

5 

6 

8 

1 

10 

Investment  Option  C 

1 

5 

10 

10 

10 

10 

Goals  Method 

Table  5.7  presents  illustrative  results  for  the  Goals  method.  Values  that  meet  or  exceed  the 
goal  value  are  highlighted  in  green.  The  table  also  shows  the  calculation  of  the  submeasure 
and  measure  scores  and  the  color  of  the  measure-score  cell  for  each  measure  that  corresponds 
to  the  color  scheme  on  the  Summary  sheet. 


11  Optimization  is  much  easier  when  the  functions  involved  are  convex.  In  the  simple  case  of  a  function  F(x),  convexity  in 
an  interval  means  that  the  derivative  of  F(x)  is  monotonically  nondecreasing.  PAT-MD  scoring  functions  violate  that  condi¬ 
tion  (see  Figure  5.2). 
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Since  the  weights  of  the  measures  are  equal,  the  effectiveness  score  is  simply  the  aver¬ 
age  of  the  two  measure  scores.  Although  Option  C  reaches  only  one  goal  in  measure  Mp  that 
submeasure  (S/3)  is  weighted  twice  as  much  as  the  other  two,  hence  it  has  the  same  measure 
as  Option  A  for  measure  M p  which  reaches  two  goals  in  measure  Mr  Since  Option  C  also 
reaches  all  of  the  goals  on  measure  M2,  it  has  the  highest  effectiveness  score  under  this  scor¬ 
ing  method. 


Table  5.7 

Illustrative  Results  for  the  Goals  Method 


Measure 

Ml 

Ml 

Ml 

M2 

M2 

M2 

Submeasure 

S1,1 

SI, 2 

SI, 3 

S2,1 

S2,2 

S2,3 

Submeasure  Weight 

0.25 

0.25 

0.5 

0.25 

0.25 

0.5 

Goal  Value 

7 

7 

7 

7 

7 

7 

Goal  Submeasure  Score 

1 

1 

1 

1 

1 

1 

Threshold  Value 

2 

2 

2 

2 

2 

2 

Threshold  Submeasure  Score 

0 

0 

0 

0 

0 

0 

Measure  1  Score 

Measure  2  Score 

Effective  ness  Score 

Investment  Option  A 

10 

10 

5 

5 

5 

5 

Submeasure  Score  (Weighted) 

0.25 

0.25 

0 

0.5 

0 

0 

0 

(0.5  +  0)/2  =  0.25 

Investment  Option  B 

7 

5 

6 

8 

1 

10 

Submeasure  Score  (Weighted) 

0.25 

0 

0 

0.25 

0.25 

0 

0.5 

0.75 

(0.25  +  0.75)/2  =  0.5 

Investment  Option  C 

1 

5 

10 

10 

10 

10 

Submeasure  Score  (Weighted) 

0 

0 

0.5 

0.5 

0.25 

0.25 

0.5 

(0.5  +  1)/2  =  0.75 

Thresholds  Method 

Table  5.8  presents  illustrative  results  for  the  Thresholds  method.  Values  that  meet  or  exceed 
the  goal  value  are  highlighted  in  green,  values  that  meet  or  exceed  the  threshold  value  but 
not  the  goal  value  are  highlighted  in  yellow,  and  values  that  fail  to  meet  the  threshold  value 
are  highlighted  in  red.  In  addition,  the  table  shows  the  calculation  of  the  measure  scores  and 
the  effectiveness  score  for  each  investment  option.  The  color  of  the  Summary  Score  cell  for 
each  measure  corresponds  to  the  color  on  the  Summary  sheet  for  the  Thresholds  method. 
The  effectiveness  score  (assuming  the  measures  have  equal  weight)  for  each  investment  op¬ 
tion  appears  in  the  rightmost  column.  In  the  Submeasure  Score  rows,  xxx  means  that  the 
submeasure  has  failed  to  reach  the  threshold  value,  so  the  measure  score  for  the  entire  mea¬ 
sure  is  0. 


Table  5.8 

Illustrative  Results  for  the  Thresholds  Method 


Measure 

Ml 

Ml 

Ml 

M2 

M2 

M2 

Submeasure 

S1,1 

SI, 2 

SI, 3 

S2,1 

S2,2 

S2,3 

Submeasure  Weight 

0.25 

0.25 

0.5 

0.25 

0.25 

0.5 

Goal  Value 

7 

7 

7 

7 

7 

7 

Goal  Submeasure  Score 

1 

1 

1 

1 

1 

1 

Threshold  Value 

2 

2 

2 

2 

2 

2 

Threshold  Submeasure  Score 

0.5 

0.5 

0.5 

0.5 

0.5 

0.5 

Measure  1  Score 

Measure  2  Score 

Effectiveness  Score 

Investment  Option  A 

10 

10 

5 

5 

5 

5 

Submeasure  Score  (Weighted) 

0.25 

0.25 

0.3 

0.15 

0.15 

0.3 

0.6 

(0.8  +  0.6)/2  =  0.7 

Investment  Option  B 

7 

5 

6 

8 

1 

Submeasure  Score  (Weighted) 

0.25 

0.15 

0.4  | 

0.25 

XXX 

0.5  | 

0  (Failure) 

(0.8  +  0)/2  =  0.4 

Investment  Option  C 
Submeasure  Score  (Weighted) 


5 

0.15 


10 


10 


10 


0.25  0.25 


(0  +  1)/2  =  0.5 
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If  an  investment  option  reaches  the  goal,  the  weighted  submeasure  score  is  1  times 
the  submeasure  weight.  If  the  raw  value  is  between  the  threshold  and  the  goal — say,  5 — then 
since  5  is  60  percent  of  the  way  from  the  threshold  (2)  to  the  goal  (7),  the  unweighted  sub¬ 
measure  score  would  be  0.6,  which  is  then  multiplied  by  the  submeasure  weight  to  get  the 
weighted  submeasure  score.  In  this  case,  since  Option  A  did  not  fail  to  reach  any  of  the 
threshold  values,  it  has  the  highest  effectiveness  score,  even  though  Option  C  meets  four  out 
of  six  goals. 

Weak  Thresholds  Method 

Table  5.9  presents  illustrative  results  for  the  Weak  Thresholds  method.  Values  that  meet  or 
exceed  the  goal  value  are  highlighted  in  green,  values  that  meet  or  exceed  the  threshold  value 
but  not  the  goal  value  are  highlighted  in  yellow,  and  values  that  fail  to  meet  the  threshold 
value  are  highlighted  in  red.  The  table  also  shows  the  calculation  of  the  measure  scores  and 
the  effectiveness  score  for  each  investment  option.  The  color  of  the  Summary  Score  cell  for 
each  measure  corresponds  to  the  color  on  the  Summary  sheet  for  the  Weak  Thresholds 
method.  The  effectiveness  score  (assuming  the  measures  have  equal  weight)  for  both  mea¬ 
sures  appears  in  the  rightmost  column.  Note  that  even  though  Option  C  fails  for  submeasure 
Sj  p  reaching  the  goal  value  on  four  of  six  submeasures  makes  its  effectiveness  greater  than 
that  of  the  other  two  options. 


Table  5.9 

Illustrative  Results  for  the  Weak  Thresholds  Method 


Measure 

Ml 

Ml 

Ml 

M2 

M2 

M2 

Submeasure 

S1,1 

SI, 2 

SI, 3 

S2,1 

S2,2 

S2,3 

Submeasure  Weight 

0.25 

0.25 

0.5 

0.25 

0.25 

0.5 

Goal  Value 

7 

7 

7 

7 

7 

7 

Goal  Submeasure  Score 

1 

1 

1 

1 

1 

1 

Threshold  Value 

2 

2 

2 

2 

2 

2 

Threshold  Submeasure  Score 

0 

0 

0 

0 

0 

0 

Measure  1  Score 

Measure  2  Score 

Effectiveness  Score 

Investment  Option  A 

10 

10 

5 

5 

5 

5 

Submeasure  Score  (Weighted) 

0.25 

0.25 

0.3 

0.15 

0.15 

0.3 

0.6 

(0.8  +  0.6)/2  =  0.7 

Investment  Option  B 

5 

6 

8 

1  10 

Submeasure  Score  (Weighted) 

0.25 

0.15 

0.4 

0.25 

0 

0.5 

0.75 

(0.8  +  0.75)/2  =  0.775 

Investment  Option  C 

5 

10  c 

10 

10 

10 

Submeasure  Score  (Weighted) 

0 

0.15 

0.5  0.65 

0.25 

0.25 

0.5 

(0.65  +  1)/2  =  0.825 

Weakest  Link  Method 

Table  5.10  presents  illustrative  results  for  the  Weakest  Link  method.  Values  that  meet  or  ex¬ 
ceed  the  goal  value  are  highlighted  in  green,  values  that  meet  or  exceed  the  threshold  value 
but  not  the  goal  value  are  highlighted  in  yellow,  and  values  that  fail  to  meet  the  threshold 
value  are  highlighted  in  red.  The  table  also  shows  the  (unweighted)  submeasure  score,  the 
measure  score  (the  minimum  of  the  submeasure  scores  for  each  measure),  and  the  effective¬ 
ness  score  (the  minimum  of  the  measure  scores).  Note  that  the  weights  of  the  submeasures 
do  not  apply  here.  The  color  of  the  effectiveness-score  cell  for  each  measure  corresponds  to 
the  color  on  the  Summary  sheet  for  the  Weakest  Link  method.  Since  Option  A  is  the  only 
option  that  does  not  fail  on  any  submeasure,  it  has  the  highest  effectiveness  score  under  this 
scoring  method. 
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Table  5.10 

Illustrative  Results  for  the  Weakest  Link  Method 


Measure 

Ml 

Ml 

Ml 

M2 

M2 

M2 

Submeasure 

S1,1 

SI, 2 

SI, 3 

S2,1 

S2,2 

S2,3 

Submeasure  Weight 

0.25 

0.25 

0.5 

0.25 

0.25 

0.5 

Goal  Value 

7 

7 

7 

7 

7 

7 

Goal  Submeasure  Score 

1 

1 

1 

1 

1 

1 

Threshold  Value 

2 

2 

2 

2 

2 

2 

Threshold  Submeasure  Score 

0 

0 

0 

0 

0 

0 

Measuie  1  Score 

Measure  2  Score 

Effectiveness  Score 

Investment  Option  A 

10 

10 

5 

5 

5 

5 

Submeas.  Score  (Unweighted) 

1 

1 

0.6 

0.6 

0.4 

0.4 

0.4 

0.4 

0.4 

Investment  Option  B 

7 

5 

6 

8 

10 

Submeas.  Score  (Unweighted) 

1 

0.4 

0.6 

0.4 

1 

0 

1 

0 

Investment  Option  C 

5 

10 

10 

10 

10 

Submeas.  Score  (Unweighted) 

0 

0.4 

1 

1 

1 

1 

0 

Rankings  Method 

In  this  example,  the  summary  and  effectiveness  weights  are  identical  (the  third  submeasures 
of  each  measure  have  twice  the  weight  of  the  other  two),  as  shown  in  Table  5.11. 

The  average  rank  and  measure-score  calculations  are  compressed  slightly,  and  all  val¬ 
ues  are  rounded  to  two  decimal  places.  For  the  measure  score,  the  sum  of  the  submeasure 
scores  is  divided  by  the  product  of  the  maximum  submeasure  score  per  measure  (2)  and  the 
sum  of  the  weights  (1).  The  effectiveness  value  is  simply  the  average  of  the  measure  scores. 
Because  Option  C  has  the  best  value  for  four  of  six  submeasures,  it  is  clearly  the  best  option 
under  the  Rankings  method. 


Table  5.11 

Illustrative  Results  for  the  Rankings  Method 


Measure 

Ml 

Ml 

Ml 

M2 

M2 

M2 

Submeasure 

81,1 

SI, 2 

SI, 3 

S2,1 

S2,2 

S2,3 

Submeasure  Weight 

0.25 

0.25 

0.5 

0.25 

0.25 

0.5 

Avq  Rank  Ideas  Score 

Avq  Rank/Meas  Score 

Effectiveness  Calc 

Investment  Option  A 

10 

10 

5 

5 

5 

5 

Rank  *  Weight 

1*0.25 

1*025 

3*0.5 

0  25  *  0  25  +  1  5  =2 

3*0.25 

2*025 

3*0.5 

0  75  +05  +  1  5  =  275 

Submeasure  Score  *  Weight 

2*0.25 

2*025 

0’0.5 

(0.5  +0.5  +0)/2  =  0  5 

0*0.25 

1*025 

0*0.5 

IP  +  0.25  +0)/2  =  0.13 

(0.5  ♦  0.13)  12  =  0.32 

Investment  Option  B 

7 

5 

6 

8 

i 

10 

Rank  *  Weight 

2*0.25 

2*025 

2*0.5 

05+05+1=2 

2*0.25 

3*025 

1*0.5 

0  5+0  75  +0.5=  1  75 

Submeasure  Score  *  Weight 

1*0.25 

0.5*0.25 

1*0.5 

(0  25  +  0  125  +  0  5)/2  =  0  44 

1*0.25 

0*025 

2*0.5 

|p25+0  +  1)/2  =  063 

(0.44  +  0.63)  (2-  0.54 

Investment  Option  C 

i 

5 

10 

10 

10 

10 

Rank  *  Weight 

3*0.25 

2*025 

1*0.5 

0  75  +0.5  +0.5  =  1  75 

1*0.25 

1*025 

1*0.5 

0  25+0  25  +05=1 

Submeasure  Score  *  Weight 

1*0.25 

0.5*0.25 

2*0.5 

(0  25  +  0  125  +  1^2  =  0  69 

2*0.25 

2*025 

2*0.5 

(0  5+05  +  iy2=  1 

(0.69  +  1)/2  =  0.85 

Marginal  and  Chunky  Marginal  Analysis 

Introduction 

In  marginal  analysis,  small  changes  to  key  variables  in  a  system  are  considered  one  at  a  time, 
and  the  resulting  changes  to  the  outputs  are  observed.  Marginal  analysis  is  often  used  to  find 
which  variables  are  most  responsible  for  affecting  outcomes.  When  the  key  variables  are  in¬ 
vestments,  marginal  analysis  helps  to  determine  where  “the  next  dollar”  (or  million  dollars) 
should  be  allocated  (or  where  it  should  be  removed)  in  order  to  maximize  the  capability  of 
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the  resulting  collection  of  investments.  Ideally,  the  small  changes  in  each  variable  are  equiva¬ 
lent  in  magnitude,  enabling  relevant  comparisons  between  options.  Although  marginal 
analysis  can  be  used  for  a  variety  of  systems,  we  will  restrict  our  discussion  to  the  marginal 
analysis  of  investments. 

Marginal  analysis  is  not  appropriate  for  all  situations.  For  example,  small  deviations 
in  investments  may  have  absolutely  no  effect  on  the  outputs  of  a  system,  particularly  when 
investments  are  tied  to  purchases  of  discrete  objects.  An  extra  dollar  invested  in  the  acquisi¬ 
tion  of  a  radar  system  has  absolutely  no  impact  when  a  single  component  costs  thousands  or 
millions  of  dollars.  Similarly,  investments  in  systems  that  require  a  large  buy-in  before  they 
become  effective  have  no  impact  until  the  buy-in  is  reached,  at  which  point  there  may  be  a 
large  discontinuity  in  capability. 

Conversely,  small  deviations  in  investments  may  have  a  disproportionate  effect  on 
the  outputs  of  the  system.  Budgeting  90  percent  of  the  cost  to  acquire  a  missile  does  not  re¬ 
sult  in  the  acquisition  of  90  percent  of  a  missile.  Likewise,  small  reductions  in  investment  do 
not  lead  to  proportionate  reductions  in  system  capability.  The  analysis  of  small  deductions  in 
investments  to  acquire  missiles,  radars,  and  so  forth  is  potentially  misleading,  as  it  tends  to 
highlight  the  systems  whose  removal  has  the  least  impact  on  overall  capability.  However,  this 
analysis  does  not  take  into  account  how  much  those  less-effective  systems  actually  cost. 

Marginal  analysis  tends  to  be  more  meaningful  when  the  options  that  result  from 
constant  deductions  to  each  investment  are  really  of  equal  value.  Consider  a  situation  where 
$16  million  is  spent  to  acquire  three  different  types  of  missiles  and  each  individual  missile 
costs  $1  million.  Assume  further  that  the  current  plan  is  to  purchase  one  missile  of  the  first 
type,  five  of  the  second  type,  and  ten  of  the  third  type.  Three  equal-cost  investment  options 
that  could  result  from  a  $1  million  cut  in  funding  correspond  to  not  purchasing  one  missile 
of  each  type.  In  contrast,  consider  a  situation  where  $16  million  is  spent  in  the  acquisition  of 
three  missiles,  one  costing  $1  million,  one  costing  $5  million,  and  one  costing  $10  million. 
A  cut  of  $1  million  from  any  of  the  three  purchases  results  in  not  getting  the  missile,  so  this 
cut  actually  results  in  a  comparison  of  three  cases  in  which  $6  million,  $11  million,  and  $15 
million  are  spent  to  acquire  two  of  the  three  types  of  missiles. 

This  suggests  a  variant  of  marginal  analysis  in  which  the  changes  to  the  current  set  of 
investments  represent  the  removal  (or  addition)  of  whole  purchases.  As  with  marginal  analy¬ 
sis,  these  purchases  are  removed  individually  from  the  current  investment,  with  each  possible 
removal  defining  a  new  investment  option.  The  costs  of  the  resulting  cases  provide  another 
measure  that  can  be  used  as  a  basis  of  comparison.  Often  the  important  question  is  not 
“Where  do  we  spend  the  next  dollar?”  but  rather  “How  do  we  invest  this  extra  $50  million?” 
(Pessimists  or  realists  will  suggest  that  the  question  asked  is  more  often  “How  do  we  handle 
this  $50  million  budget  cut?”)  This  chunky  marginal  analysis  method  will  be  described  in 
more  detail  in  the  next  section,  as  will  the  application  of  PAT-MD  to  assist  with  it. 

Chunky  Marginal  Analysis 

We  consider  as  a  base  case  a  fixed  collection  of  BMDS  investments  over  time  in  both  R&D 
and  acquisitions.  We  generate  a  collection  of  investment  options  by  taking  steps  away  from 
the  base  case.  These  steps  are  large-scale  additions  to  (or  subtractions  from)  the  base  case. 
Some  examples  would  be  the  cancellation  of  R&D  on  a  particular  program  or  a  reduced  (or 
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increased)  acquisition  of  a  radar  system  or  interceptor.  The  steps  should  not  be  so  small  that 
it  is  impossible  to  discern  between  the  base  case  and  the  BMDS  associated  with  the  new  in¬ 
vestment  option,  but  they  should  not  be  so  large  that  they  could  be  decomposed  into  a  col¬ 
lection  of  smaller,  meaningful  steps.  Thus,  a  step  should  not  consist  of  the  cancellation  of 
three  unrelated  programs;  rather,  three  new  options  should  be  defined  by  the  cancellation  of 
each  individual  program.  In  addition,  a  step  should  be  maximal  in  the  sense  that  any  pro¬ 
gram  that  is  made  unnecessary  (or  necessary)  by  the  cancellation  (or  addition)  of  one  pro¬ 
gram  should  also  be  cut  (or  included)  along  with  that  program.  For  example,  cutting  the  de¬ 
velopment  of  a  radar  platform  should  also  cut  the  development  of  any  battle-management 
suite  associated  with  it  (unless  the  suite  can  also  be  used  for  other  radar  systems  under  devel¬ 
opment). 

Once  these  steps  are  defined,  each  investment  option  consists  of  the  base  case  and 
one  (or  more)  of  these  steps.  As  the  number  of  steps  increases,  the  number  of  possible  in¬ 
vestment  options  increases  exponentially.  Analysis  should  probably  be  restricted  to  options 
that  are  at  most  a  fixed  number  of  steps  away  from  the  base  case.  With  n  possible  steps,  there 
are  roughly  n  12  options  that  are  at  most  two  steps  away  from  the  base  case  and  about  n3 16 
options  that  are  at  most  three  steps  away  from  the  base  case.  The  number  of  investment  op¬ 
tions  under  consideration  should  also  be  tempered  by  the  ability  to  determine  the  costs  asso¬ 
ciated  with  each,  as  well  as  the  ability  to  analyze  the  BMDS  that  results  from  each.  PAT-MD 
can  store  thousands  of  investment  cases  and  can  display  some  or  all  of  those  cases  on  its  out¬ 
put  sheets. 

The  output  sheet  that  is  perhaps  most  useful  in  assisting  with  chunky  marginal  analy¬ 
sis  is  the  Scatter  Plot  sheet,  where  the  user  can  select  the  x-axis  from  a  collection  of  cost  met¬ 
rics  (over  various  time  periods,  restrained  to  R&D  investment  or  deployment  investment 
only,  etc.)  and  up  to  two  different  evaluation  metrics  as  the  y-axis.  Investment  options  are 
plotted  on  the  display  as  points  of  different  colors  (and  different  shapes  if  more  than  one 
measure  is  displayed  on  the  y-axis).  This  is  particularly  useful  for  determining  which  invest¬ 
ment  option  is  best  (and  the  corresponding  programs  that  should  be  cut)  if  the  budget  is  re¬ 
duced  from  the  amount  allocated  in  the  base  case. 

Consider  a  notional  base  case  that  consists  of  investments  in  three  ESGs:  one  boost 
phase,  one  midcourse,  and  one  terminal  phase  (based  in  the  United  States).  There  are  three 
obvious  steps  away  from  this  base  case,  namely,  the  cancellation  of  programs  specific  to  each 
phase  of  the  defense  (defense  systems  from  different  phases  may  share  tracking  systems,  so  a 
cut  of  a  particular  phase  of  the  defense  would  not  necessarily  eliminate  all  programs  associ¬ 
ated  with  that  phase).  Because  the  number  of  steps  is  so  small,  there  is  no  reason  not  to  con¬ 
sider  the  eight  investment  options  corresponding  to  all  the  subsets  of  the  phases  of  defense 
that  can  be  implemented. 

Next,  consider  the  probabilities  of  engagement  versus  a  single  attacker  shown  in 
Table  5.12  for  four  different  scenarios:  HD,  HD  with  no  boost-phase  access,  HD  with 
advanced  countermeasures,  and  DODF/DOFA. 

Assuming  that  the  performances  of  the  systems  in  each  phase  are  independent  of  one 
another,  we  can  compute  the  probability  that  a  single  missile  is  intercepted  for  each  of  the 
eight  investment  options,  as  well  as  for  each  of  the  four  scenarios,  which  can  be  thought  of  as 
measures.  If  we  set  a  threshold  value  of  0.5  and  a  goal  value  of  0.8  for  each  scenario,  we  get 
the  color  scheme  shown  in  Table  5.13  (which  would  appear  on  the  Details  sheet)  under  the 
default  (Thresholds)  scoring  method. 
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Table  5.12 

Contrived  Data  for  Illustrative  Problem 


Probability  of  Engagement 

Cost 

(billions  of  $) 

Phase  of  Defense 

HD 

No  Boost  Phase 

Advanced 

Countermeasures 

DODF/DOFA 

Boost  phase  (B) 

0.7 

0.0 

0.5 

0.7 

9 

Midcourse  (M) 

0.6 

0.6 

0.4 

0.6 

6 

Terminal  (T) 

0.5 

0.5 

0.3 

0 

3 

Table  5.13 

Performance  of  the  Investment  Options  in  Different  Scenarios 

Probability  of  Engagement 

Cost 

Phase  of  Defense  HD  No  Boost  Phase  Advanced  CMs  DODF/DOFA  (billions  of  $) 


B  +  M  +  T 
B  +  M 
B  +  T 
M  +  T 
B  only 
M  only 
T  only 
None 


18 

15 

12 

9 

9 

6 

3 

0 


Treating  the  individual  scenarios  as  measures  (or  to  be  accurate,  treating  each  column 
as  a  measure  consisting  of  a  single  submeasure),  we  can  calculate  the  effectiveness  of  each  in¬ 
vestment  option  for  the  Thresholds  scoring  method.  Assuming  all  scenarios  are  equally 
weighted,  with  a  submeasure  score  of  0.5  for  reaching  the  threshold  value  and  a  value  of  1 
for  reaching  the  goal  value  for  each  scenario,  the  effectiveness  score  of  each  investment  op¬ 
tion  is  as  shown  in  Table  5.14.  The  numbers  in  brackets  are  the  submeasure  scores  for  each 
submeasure. 

Table  5.14 

Costs  and  Effectiveness  Comparisons:  Equal  Emphasis  on  All  Scenarios 


Probability  of  Engagement  and  Effectiveness  Score 


Effectiveness  Cost 

Phase  of  Defense  HD  No  Boost  Phase  Advanced  CMs  DODF/DOFA  (Sum/4)  (billions  of  $) 


B  +  M  +  T 
B  +  M 
B  +  T 
M  +  T 
B  only 
M  only 
T  only 
None 


0.94  [1] 

0.8  [1] 

0.79  [0.98] 

0.88  [1] 

0.996 

0.88  [1] 

0.6  [0.67] 

0.7  [0.83] 

0.88  [1] 

0.875 

0.85  [1] 

0.5  [0.5] 

0.65  [0.75] 

0.7  [0.83] 

0.771 

0.8  [1] 

0.8  [1] 

0.58  [0.64] 

0.6  [0.67] 

0.7 

0.542 

0.5 

0.25 

0 


18 

15 

12 

9 

9 

6 

3 

0 
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To  put  greater  emphasis  on  dealing  with  a  peer  threat  that  can  deny  boost-phase  ac¬ 
cess  and  implement  advanced  countermeasures,  we  can  weight  those  two  scenarios  twice  as 
much  as  the  other  two  (with  the  numbers  in  brackets  now  representing  the  submeasure/ 
measure  score  times  the  weight  of  the  measure).  The  effectiveness  of  each  investment  option 
for  this  case  is  shown  in  Table  5.15. 

The  Scatter  Plot  sheet  can  be  used  to  display  the  effectiveness  of  each  investment  case 
for  both  perspectives,  with  the  cost  of  the  investment  option  as  the  x-axis  (see  Figure  5.3). 

Table  5.15 

Costs  and  Effectiveness  Comparisons:  Extra  Emphasis  on  Peer  Threat 


Probability  of  Engagement  and  Effectiveness  Score 


Scaled 

Effectiveness  Cost 

Phase  of  Defense  HD  No  Boost  Phase  Advanced  CMs  DODF/DOFA  (Sum/6)  (billions  of  $) 


B  +  M  +  T 
B  +  M 
B  +  T 
M  +  T 
B  only 
M  only 
T  only 
None 


0.94  [1] 

0.8  [2] 

0.79  [1.96] 

0.88  [1] 

0.994 

0.88  [1] 

0.6  [1.33] 

0.7  [1.67] 

0.88  [1] 

0.833 

0.85  [1] 

0.5  [1] 

0.65  [1.5] 

0.7  [0.83] 

0.722 

0.8  [1] 

0.8  [2] 

0.58  [1.28] 

0.6  [0.67] 

0.822 

0.444 

0.444 

0.25 

0 


18 

15 

12 

9 

9 

6 

3 

0 


Figure  5.3 

Cost-Effectiveness  Comparisons  for  Two  Perspectives 
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The  scatter  plot  view  provides  a  straightforward  comparison  of  the  investment  op¬ 
tions  under  both  perspectives.  The  chart  is  unconventional,  however,  so  reading  it  requires 
some  instruction.  Suppose  that  the  budget  must  be  cut  to  a  level  of  $9  billion  (a  cut  of  $9 
billion).  At  that  level  (x-axis),  the  best  option  from  the  first  perspective  (squares)  is  M  +  T 
(elimination  of  boost  phase).  That  is,  the  topmost  square  has  the  color  associated  with  the 
M  +  T  option  in  the  legend  box  to  the  right.  When  PAT-MD  is  being  used  live,  we  would 
identify  that  topmost  square  by  merely  passing  the  mouse  over  it  and  seeing  its  name  pop  up. 

If  we  are  more  concerned  about  the  peer  threat  to  the  homeland,  we  look  at  the  effec¬ 
tiveness  scores  shown  in  the  circles  and  find  that  the  same  investment  option  is  best.  That 
certainly  might  not  have  been  the  case.  For  example,  consider  budget  cuts  of  $3  billion,  $6 
billion,  and  $9  billion.  As  indicated  in  Figure  5.3,  in  this  contrived  example,  the  terminal 
phase,  midcourse,  and  boost-phase  ESG  should  be  canceled,  respectively,  to  maintain  the 
greatest  effectiveness  for  the  equal-weighting  perspective.  Although  the  boost-phase  ESG 
costs  as  much  as  retaining  the  midcourse  and  terminal  phase  ESGs,  the  capability  is  much 
lower.  This  is  amplified  under  the  peer  emphasis  perspective  (the  squares),  when  the  effec¬ 
tiveness  score  of  the  midcourse-ESG-only  and  the  boost-phase-ESG-only  options  are  nearly 
equivalent.  For  budget  cuts  between  $3  billion  and  $9  billion,  the  cancellation  of  the  boost- 
ph  ase-ESG  option  has  a  higher  effectiveness  than  does  the  cancellation  of  the  less-expensive 
midcourse-ESG  option.  Thus,  depending  on  the  perspective,  different  priorities  are  placed 
on  the  retention  of  different  phases  of  defense. 

In  general,  the  steps  will  be  much  smaller  than  the  cancellation  of  entire  ESGs. 
Cancellations  (or  delays)  of  single  components  of  an  ESG  should  be  the  options  under  con¬ 
sideration.  Chunky  marginal  analysis  is  particularly  illuminating  when  components  under 
consideration  are  used  across  multiple  ESGs,  assisting  with  large-scale  investment  decisions. 


6.  Concluding  Observations 


Purpose  and  Function  of  PAT-MD 

PAT-MD  is  not  a  model;  rather,  it  is  an  “empty-vessel”  tool.  Its  purpose  is  to  help  frame, 
manipulate,  and  present  multifaceted  information  to  decisionmakers,  particularly  with 
capabilities-based  planning  in  mind.  It  is  designed  to  work  in  parallel  with  a  capabilities 
model  (CAMMD)  for  the  missile-defense  problem  and  to  use  diverse  information  such  as 
structured  expert  judgment  and  management  philosophy  (e.g.,  whether  decisionmakers  in  a 
particular  meeting  or  study  are  seeking  hard-nosed  “requirements-based”  system  assessments 
or  more  of  a  progress  assessment). 


Limitations  and  Cautions 

At  first  glance,  PAT-MD  (or  the  related  tool,  DynaRank)  may  appear  to  be  useful  for  opti¬ 
mizing  resource  allocation  mathematically.  The  mechanism  for  calculating  cost-effectiveness 
might  seem  to  imply  that  the  objective  is  or  should  be  to  maximize  that  quantity.  That 
would  be  a  misreading  of  our  intentions,  and  indeed  of  the  philosophy  underlying  our  ap¬ 
proach.  The  most  important  outputs  of  PAT-MD  are  (1)  the  portfolio-style  scorecards  in 
which  alternative  investment  options  are  assessed  simultaneously  by  a  number  of  very  differ¬ 
ent  measures  and  costs;  and  (2)  the  next  layer  of  scorecard  detail  to  which  the  viewer  is  able 
to  zoom  in  order  to  understand  the  basis  of  the  color-coded  summary  assessment  and  to 
change  higher-level  assumptions  or  priorities  that  affect  the  assessments. 

We  believe  that  further  aggregation  to  a  single  number,  as  in  cost-effectiveness  cal¬ 
culations,  should  be  deferred  to  a  kind  of  refinement  stage,  a  stage  in  which  the  user  is  tidy¬ 
ing  up  and  thinking  about  communicating  results  of  decisions.  Cost-effectiveness  calcula¬ 
tions  depend  sensitively  on  the  assumptions  and  priorities  that  go  into  them,  which  is 
precisely  what  decisionmakers  are  paid  for.  They  are  the  people  responsible  for  worrying 
about,  say,  the  “balance”  of  a  portfolio  across  missions,  the  extent  of  risk  to  be  taken,  and  the 
ways  in  which  risk  can  be  managed.  To  do  so,  they  need  to  reason  at  the  portfolio  level,  not 
at  the  level  of  merely  comparing  cost-effectiveness  numbers. 

It  follows  from  this  that  PAT-MD  should  not  be  seen  as  a  tool  for  mathematical  op¬ 
timization,  much  less,  simple  do-it-by-the-numbers  optimization.  PAT-MD  provides  a 
number  of  ways  to  assess  alternatives  and  some  useful,  albeit  limited,  mechanisms  for  ex¬ 
ploring  the  consequences  of  alternative  assumptions  and  priorities,  but  that  is  very  different 
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from  optimization.  PAT-MD  is  intended  to  facilitate  finding  strategies  that  are  flexible, 
adaptive,  and  robust. 1 

PAT-MD  as  Software 

This  first  version  of  PAT-MD  is  not  industrial-strength  software;  rather,  it  is  a  tool  devel¬ 
oped  for  analysis  in  certain  contexts.  It  has  been  tested  to  a  significant  degree  but  un¬ 
doubtedly  has  residual  problems  (both  bugs  and  features).  Users  are  encouraged  to  contact 
PAT-MD’s  programmer,  Paul  Dreyer,  at  dreyer@rand.org,  if  they  discover  mistakes  or  have 
questions  about  the  operation  of  PAT-MD. 

PAT-MD  checks  for  most  mistakes  that  a  user  can  make  (for  example,  entering  an 
investment  option  or  measure  in  the  Summary  table  that  does  not  exist  on  the  Data  Entry 
sheet).  However,  it  has  not  been  “gorilla-proofed.”  For  example,  the  Dropdown  sheet,  which 
stores  all  of  the  entries  for  the  dropdown  menus,  cannot  be  locked  to  prevent  user  manipula¬ 
tion  as  it  is  frequently  updated  by  PAT-MD.  Tinkering  with  the  entries  on  that  sheet  would 
lead  to  errors  in  PAT-MD.  Ekcause  of  thisS  we  encourage  users  to  keep  a  clean  copy  of  the 
PATQ V D  template  available  in  the  unlikely  event  that  something  occurs  to  make  the  software  unG 
usable: 

The  proper  use  of  PAT-MD  also  requires  considerable  discipline.  With  the  Template 
Huilder  sheet,  we  have  attempted  to  simplify  much  of  the  initial  setup  of  a  portfolio  view,  but 
other  operations  require  consistency.  For  example,  if  one  edits  the  values  on  the  Details  sheet, 
it  is  necessary  to  click  on  the  Modify  Data  Entries  button  in  order  for  those  changes  to 
propagate  through  the  rest  of  the  tool.  Similarly,  the  Scatter  Plot  and  Spider  Chart  sheets  are 
not  automatically  updated  when  changes  are  made  to  the  Data  Entry  sheet,  so  it  is  necessary 
to  click  on  the  Generate  Scatter  Plot  and  Generate  Spider  Chart  buttons  on  the  applicable 
sheets  to  see  how  the  changes  in  the  data  have  changed  the  outputs. 

PAT-MD  allows  considerable  flexibility  in  what  users  can  change  without  having  to 
regenerate  the  portfolio  view,  as  described  above.  Weights  on  measures  and  submeasures, 
threshold  and  goal  values,  scoring  methods,  and  data  values  can  be  changed  easily.  However, 
one  should  be  cautious  when  adding  or  removing  ESGs,  BMDS  components,  investment 
options,  measures,  or  submeasures,  as  they  appear  in  several  places  throughout  PAT-MD 
(and  in  some  cases,  in  very  particular  order).  Again,  we  caution  that  an  unblemished  copy  of 
the  PAT-MD  template  should  be  kept  along  with  any  portfolio  tools  developed,  in  order  to 
be  able  to  easily  reconstruct  a  portfolio  view  if  several  new  objects  need  to  be  added. 

The  only  constraints  on  the  numbers  of  investment  options,  measures  and  submea¬ 
sures,  etc.,  are  those  imposed  by  the  ability  of  an  EXCEL  spreadsheet  to  hold  all  of  the  input 
data.  An  EXCEL  spreadsheet  can  have  up  to  65,536  rows  and  256  columns,  so  it  can  contain 
up  to  255  submeasures,  251  ESGs,  and  255  perspectives,  and  the  only  constraint  on  the 
number  of  investment  options  is  that  the  cost  data  must  fit  on  a  single  spreadsheet  (so  the 
product  of  the  number  of  BMDS  components  and  investment  options  cannot  exceed 
65,536).  This  should  provide  more  than  enough  room  to  store  all  investment  options  of 
interest. 


1  This  emphasis  in  RAND  work  has  been  articulated  in  a  number  of  monographs  (Davis,  1994;  Davis,  2002),  including 
some  concerned  with  very  long-range  planning  (Lempert,  Popper,  and  Bankes,  2003).  It  is  quite  consistent  with  the  ap¬ 
proach  to  capabilities-based  planning  first  emphasized  by  DoD  in  the  2001  Quadrennial  Defense  Review  (Department  of 
Defense,  2001)  and  subsequently  reinforced. 
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The  Importance  of  the  Measures,  Submeasures,  and  Methods  Used 

As  with  all  scorecard  methods,  considerable  care  must  be  taken  in  the  development  of  the 
measure/submeasure  structure  for  any  analysis  using  PAT-MD.  Similar  care  should  be  taken 
with  selection  of  the  scoring  method  and  the  parameters  and  weights  for  the  scoring  func¬ 
tions  and  effectiveness  calculations.  If  PAT-MD  used  only  linear  weighted  sums,  the  meth¬ 
odology  would  be  simpler,  but  it  would  be  less  satisfactory  for  systems  analysis. 

Although  we  cannot  itemize  here  all  the  considerations  that  an  analyst  should  have  in 
mind,  a  few  are  particularly  worth  mentioning: 

•  Measures  and  submeasures  chosen  should  be  adequately  complete  in  the  assessment 
they  provide. 

•  Ideally,  measures  would  be  independent,  and  the  submeasures  of  each  measure  would 
be  independent.  When  that  is  not  appropriate — i.e.,  when  correlations  exist — 
weighting  factors  should  be  chosen  so  as  to  avoid  results  being  overly  sensitive  to  a 
single  underlying  issue.2 

•  The  choice  of  scoring  and  aggregation  methods  merits  particular  thought.  If  all  of  a 
measure’s  submeasures  are  individually  critical,  the  threshold  method  may  be  called 
for.  If  all  of  the  measures  are  also  individually  critical,  the  Weakest  Link  method  may 
be  appropriate.  If  these  conditions  do  not  apply,  however,  and  it  is  more  important 
to  see  progress  than  to  flag  problems,  the  Goals  method  may  be  appropriate.  The 
Rankings  method  can  be  helpful  when,  for  one  reason  or  another,  it  is  inappropriate 
to  discuss  goals  and  thresholds. 

•  In  any  case,  it  is  essential  to  plan  for  systematic  exploration  of  the  ways  assumptions 
on  the  above  matters  affect  both  results  and  perceptions,  and  to  tune  assumptions  so 
that  they  provide  a  set  of  baseline  results  that  are  as  robust  as  possible.  For  example,  it 
would  serve  decisionmakers  poorly  if  color-coded  conclusions  were  to  change  mark¬ 
edly  when  a  few  low-level  assumptions  were  changed  slightly  (e.g.,  moving  a  goal 
from  0.89  to  0.9  should  not  change  results  dramatically). 

•  A  consequence  of  the  above  admonition  is  that  goals  and  thresholds  need  to  be  seen 
as  heuristics,  not  as  absolutes  to  be  accepted  mindlessly  as  impossible-to-question 
“data.” 

Finally,  we  note  that  many  of  these  issues  are  generic  and  there  is  a  considerable  lit¬ 
erature  dealing  with  multi-attribute  measures  and  objectives  that  discusses  approaches  to 
weighting  these  measures,  obtaining  utilities  from  individuals  or  groups,  and  the  use  of  other 
aggregation  rules.3 


2  For  example,  an  option  could  obviously  be  made  to  look  better  by  piling  on  a  number  of  measures,  each  of  which  is 
driven  by  something  accomplished  well  by  the  option.  Similarly,  an  option  may  appear  worse  than  it  probably  should  if  the 
measures  chosen  reflect  a  pure  worst-case  perspective. 

3  Some  of  these  appear  under  discussion  of  multi-attribute  utility  theory  (Keeney  and  Raiffa,  1976;  Kirkwood,  1997), 
value-focused  thinking  (Keeney,  1992;  Parnell,  2004),  and  balanced-scorecard  methods  used  in  business  (Kaplan  and 
Norton,  1996).  The  original  DynaRank  documentation  (Hillestad  and  Davis,  1998)  also  includes  some  such  discussion. 
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Next  Steps 

Over  time,  PAT-MD  will  be  improved  and  enhanced,  building  on  experience  and  use.  Sug¬ 
gestions  will  therefore  be  appreciated.  In  addition  to  correcting  any  errors  that  are  discovered 
and  improving  user-friendliness,  we  are  currently  thinking  about  at  least  the  following  possi¬ 
bilities  for  enhancement: 

•  An  additional  layer  of  hierarchy  for  zooming. 

•  Permitting  different  scoring  and  aggregation  methods  to  be  used  for  different  mea¬ 
sures  or  for  calculation  of  cost-effectiveness  rather  than  measure  scores. 

•  The  ability  to  generate  the  measure  of  cost  used  in  cost-effectiveness  as  a  linear 
weighted  sum  of  the  various  costs  provided  as  inputs. 

•  A  richer  and  more  structured  mechanism  for  exploratory  analysis,  probably  building 
on  the  alternative-perspectives  mechanism.  This  could  include  limited  mechanisms 
for  search  (e.g.,  finding  what  combinations  of  key  parameters  would  cause  a  particu¬ 
lar  option  to  be  assessed  well  or  poorly). 


Appendix  A 

Comparison  of  Features  of  PAT-MD  and  DynaRank 


What  Is  DynaRank? 

DynaRank  is  a  hierarchical  scorecard  tool  developed  by  RAND  for  portfolio  analysis 
(Hillestad  and  Davis,  1998).  It  is  very  flexible  and  allows  for  high-level  integration  of  cost 
and  performance  data  across  one  or  more  objectives.  A  sample  DynaRank  scorecard  is  shown 
in  Figure  A.l.  As  in  PAT-MD,  rows  on  the  scorecard  correspond  to  the  different  investment 
(or  strategic)  options;  columns  correspond  to  the  measures  used  to  evaluate  the  investment 
options.  DynaRank  has  a  three-level  structure  of  measures  (top,  mid-level,  and  base).  Values 
in  individual  tables  are  all  measured  on  the  same  scale,  with  increasing  values  implying 
increasing  utility. 

Figure  A.l 
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In  Figure  A.l,  the  first  column  corresponds  to  the  mid-level  measure  of  IOC,  which 
is  subordinate  to  the  top-level  measure  of  time,  while  the  fourth  column  corresponds  to  the 
base-level  measure  of  intertheater  (transport),  which  is  subordinate  to  the  mid-level  measure 
of  transport,  which  is  subordinate  to  the  top-level  measure,  resources.  The  values  of  measures 
can  be  aggregated  up  to  the  mid  level  or  top  level,  using  either  weighted  linear  sums  of  sub¬ 
ordinate  values  or  a  minimum-value  aggregation,  where  the  aggregate  value  of  a  mid-level  (or 
top-level)  measure  is  the  smallest  value  among  the  subordinate  base-level  (or  mid-level) 
measures.  The  values  in  the  cost  column  are  marginal  costs  of  the  options  relative  to  a  base 
case  (the  measure  scores  of  which  are  in  the  top  row  of  the  table).  Colors  of  the  values  range 
from  red  to  green,  based  on  a  0  to  100  scale  (this  scale  can  be  changed) . 

What  Is  the  Same  in  Both  Tools?  What  Is  Different? 

DynaRank  is  the  inspiration  for  PAT-MD,  and  both  are  written  in  Visual  Basic  with  an 
EXCEL  front  end;  both  provide  dynamic  views  of  information  (allowing  changing  of 
weights,  scoring  methods,  and  views  on  the  fly).  Both  provide  means  of  comparing  invest¬ 
ment  and/or  strategic  options  via  aggregations  of  diverse  measures  that  can  come  from  other 
models,  quantitative  assessments,  and/or  subjective  judgments.  In  addition,  both  use  color- 
coded  scorecards  to  present  information,  although  the  methodologies  underlying  how  those 
colors  are  determined  differ  somewhat  between  the  two  tools.  In  addition,  both  incorporate 
cost  data  into  their  displays  and  present  the  cost-effectiveness  of  options.  Both  provide  dis¬ 
plays  that  rank  investment  options  under  multiple  perspectives  (weights  on  measures,  etc.), 
and  both  can  display  a  subset  of  the  entire  set  of  investment  options  (which  is  useful  when 
there  are  many  options).  Fiowever,  the  two  tools  have  completely  different  code  underlying 
their  construction,  and  the  manner  in  which  they  present  results  differs  considerably.  Simi¬ 
larities  and  differences  between  DynaRank  and  PAT-MD  are  shown  in  Table  A.l. 


When  to  Use  DynaRank  and  When  to  Use  PAT-MD 

Both  tools  are  highly  flexible  in  the  ways  they  can  present  information.  If  setting  goal  and 
threshold  values  for  the  submeasures  is  reasonable  and  straightforward,  or  if  a  rankings- 
based  aggregation  method  is  appropriate  for  comparing  options,  one  should  use  PAT-MD. 
PAT-MD  also  provides  more  options  for  the  display  of  information  than  DynaRank  does, 
and  it  may  be  a  better  tool  for  the  presentation  of  results.  While  PAT-MD  can  be  used  for 
marginal  analysis,  the  user  who  is  willing  to  use  linear  aggregation  methods  to  compare 
options  will  probably  find  DynaRank  the  right  tool  for  the  job.  Currently,  PAT-MD  is  spe¬ 
cialized  for  analysis  for  MDA,  but  it  could  easily  be  modified  to  apply  to  a  wide  variety  of 
problems.  In  fact,  much  of  the  functionality  in  DynaRank  is  replicated  in  PAT-MD  in  some 
fashion.  Fiowever,  DynaRank  is  more  flexible  in  the  sense  that  it  can  be  used  as  a  scorecard 
for  any  multi-measure,  multi-objective  problem.  At  this  stage,  it  has  also  been  tested  more 
extensively  and  has  been  used  in  a  number  of  studies. 

In  summary,  we  developed  PAT-MD  for  the  MDA  application  because  it  was  impor¬ 
tant  to  incorporate  both  goals  and  thresholds  into  our  analysis,  to  have  some  new  output 
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displays,  to  have  specialized  input  sheets,  to  have  more  flexibility  in  aggregation  methods, 
and  to  introduce  new  functionality  such  as  the  IOC-to-cost-data  calculations. 

Table  A.1 

Comparison  of  DynaRank  and  PAT-MD 

DynaRank  PAT-MD 


Data 


Investment 

options 


Aggregation 

methods 


Displays 


Exploration 
across  different 
perspectives 

Computing  invest¬ 
ment  program 
from  time  goals 


Three  levels  of  measures  (top,  middle,  and 
base). 

All  data  values  have  the  same  scale  and 
imply  increasing  utility  as  values  increase. 

Can  have  multiple  datasets  and  scorecards 
per  workbook. 

Options  are  intended  to  be  marginal  steps 
away  from  a  base  case  (although  the  base 
case  can  be  empty). 

Cost  data  for  each  option  are  the  marginal 
cost  away  from  the  base  case. 

Aggregate  by  linear  weighted  sums  or 
weighted  minimum  values  (the  smallest 
subordinate  value  propagates  upward). 

Can  display  scorecards  for  any  level  of  mea¬ 
sure  with  cost  data. 

Can  have  hierarchies  of  scorecards  or  multiple 
scorecards  on  same  sheet. 

Colors  on  scorecards  are  directly  related  to 
scores  in  cells  by  a  single  aggregation 
method. 

Can  plot  cumulative  costs  and  effectiveness 
for  different  perspectives. 

Yes  (called  views  in  DynaRank). 


No. 


Two  levels  of  measures  (measures  and  sub¬ 
measures). 

Data  can  have  different  scales  and  can  have 
both  high  and  low  values  as  good  values. 

One  dataset  per  workbook. 

No  base  case  is  assumed  (although  chunky 
marginal  analysis  can  still  be  done). 

Cost  data  represent  the  cost  of  the  entire 
investment  option  (as  a  collection  of 
investments  in  components  over  time). 

Goal-  and  threshold-based,  Weakest  Link,  or 
rankings-based  aggregations. 

Can  use  different  aggregation  methods  for 
different  displays. 

Main  display  can  show  both  measure  and 
submeasure  data,  plus  cost  data. 

Can  highlight  risk  on  Summary  sheet  along 
with  submeasure  values  and  cost  data. 

Colors  on  scorecards  depend  on  aggregation 
method. 

Spider  charts,  scatter  plots,  cost  and  ESG  in¬ 
formation  graphs. 

Yes. 


Yes  (the  IOC  to  Cost  Data  sheet). 
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PAT-MD  can  be  used  to  develop  different  sets  of  displays  for  different  purposes  (e.g.,  strate¬ 
gic  planning  versus  on-the-margin  budget  scrubs).  These  displays  are  called  views }  Creating 
one  is  an  excellent  way  to  become  familiar  with  the  nuts  and  bolts  of  entering  values  into 
PAT-MD. 

Given  an  empty  PAT-MD  workbook,  a  user  can  build  a  results  display  using  what¬ 
ever  investment  options,  measures,  and  submeasures  he  or  she  desires.  Before  any  display  is 
developed,  the  user  must  generate  the  following  lists  of  objects  used  in  the  analysis  (and  the 
names  used  to  identify  them,  which  must  be  consistent  throughout  the  tool): 

1.  Investment  options.  The  options  that  are  being  evaluated  in  PAT-MD. 

2.  Measures  and  submeasures.  Criteria  on  which  the  investment  options  are  evaluated.  Each 
measure  consists  of  an  aggregation  of  submeasures;  measures  do  not  need  to  have  the 
same  submeasures. 

3.  BMDS  components.  The  investment  options  allocate  different  amounts  of  funding  over 
time  to  each  BMDS  component  under  consideration  (for  both  RDT&E  and  deploy¬ 
ment). 

4.  Engagement  sequence  groups  (ESGs).  For  an  ESC  to  be  considered  a  potential  option,  all  of 
the  BMDS  components  it  comprises  must  be  completed. 

5.  Time  frame.  The  years  the  analysis  and  cost  data  should  cover. 

Once  these  have  been  determined,  the  user  can  begin  to  fill  in  the  input  sheets.  At 
the  end  of  this  appendix,  we  describe  the  Template  Builder  worksheet,  which  simplifies  the 
setup  of  all  of  the  PAT-MD  worksheets. 


Filling  in  the  Data  Entry  Sheet 

1 .  Add  the  names  of  the  investment  options  to  the  first  column,  starting  in  the  row  below 
the  yellow  bar  (row  10). 

2.  Fill  in  the  measures  and  submeasures  information.  Starting  with  column  B,  for  each 
measure/submeasure  pair,  put  the  name  of  the  measure  in  row  1  and  the  name  of  the 
submeasure  in  row  2.  Submeasures  of  the  same  measure  must  appear  in  consecutive  col¬ 
umns.  To  include  a  risk  column  for  a  measure,  input  it  as  a  submeasure  with  the  name 
Risk. 


1  A  “view”  in  DynaRank  is  called  a  “perspective”  in  PAT-MD. 
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3.  Fill  in  the  parameters  (weights  for  effectiveness  and  summary  calculations,  threshold  and 
goal  values,  etc.)  for  each  submeasure  in  rows  3  through  8.  If  a  measure  has  a  risk  col¬ 
umn,  these  values  should  be  left  blank. 

4.  Fill  in  the  (numeric)  values  for  each  submeasure  for  each  investment  option  in  the  corre¬ 
sponding  row  and  column.  The  only  column  that  should  have  nonnumeric  values  is  the 
Risk  column. 


Filling  in  the  Cost  Data  Sheet 

The  Cost  Data  sheet  consists  of  a  collection  of  blocks  of  cost  numbers  that  represent  the 
amount  of  RDT&E  or  deployment  funding  per  BMDS  component  per  year  for  an  invest¬ 
ment  option.  These  blocks  and  how  they  are  arranged  have  a  very  particular  structure,  as 
shown  in  Figure  B.l. 

1 .  The  cell  in  the  first  row  and  column  of  each  block  denotes  whether  the  block  de¬ 
scribes  RDT&E  funding  or  deployment  funding. 

2.  The  second  row  of  the  block  has  the  name  of  the  investment  option  in  the  first  col¬ 
umn  and  the  list  of  years  in  the  time  frame  under  consideration  in  each  successive 
column. 

3.  The  successive  entries  in  the  first  column  are  the  names  of  the  BMDS  components, 
and  the  entries  in  the  cells  show  the  amount  of  funding  for  each  component  for  each 
year. 

Figure  B.l 

Block  Structure  of  Cost  Data  Sheet  for  Two  Investment  Options 
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The  resulting  block  should  have  two  more  rows  than  the  number  of  components  and 
one  more  column  than  the  number  of  years  in  the  time  frame.  The  RDT&E  block  for  the 
first  investment  option  must  have  its  upper  left-hand  corner  in  the  upper  left-hand  corner  of 
the  Cost  Data  sheet.  The  RDT&E  and  deployment  blocks  for  the  same  investment  option 
should  be  side-by-side,  separated  by  a  single  column.  A  single  row  should  separate  blocks  for 
consecutive  investment  options.  The  sheet  should  fill  2Y  +  3  columns,  where  Y  is  the  num¬ 
ber  of  years  in  the  time  frame,  and  I(C  +  3)  -  1  rows,  where  I  is  the  number  of  investment 
options  and  C  is  the  number  of  BMDS  components.  The  BMDS  components  must  be  in 
the  same  order  in  each  block. 


Filling  in  the  ESG  Data  Sheet 

1 .  The  names  of  the  BMDS  components  should  be  entered  in  column  E,  starting  in  row  2 
in  the  same  order  as  on  the  Cost  Data  sheet. 

2.  For  each  component,  the  amount  of  investment  to  complete  the  RDT&E  should  appear 
in  the  first  column. 

3.  The  B,  C,  and  D  columns  should  contain  0/1  flags  to  denote  whether  the  component  is 
intended  for  use  in  a  boost-phase,  midcourse,  or  terminal-phase  ESG.  These  flags  are 
used  to  produce  the  phase-specific  cost  tables  on  the  ESG  Table  sheet. 

4.  The  names  of  the  ESGs  should  be  placed  in  consecutive  cells  in  row  1,  starting  with  col¬ 
umn  F. 

5.  Each  cell  corresponding  to  a  BMDS  component  and  an  ESG  should  contain  a  1  if  the 
BMDS  component  is  a  part  of  the  ESG  and  a  0  (or  blank)  otherwise. 


Filling  in  the  Perspective  Cases  Sheet 

Defining  a  new  perspective  case  is  equivalent  to  changing  by  hand  the  parameters  for  the 
submeasures  on  the  Data  Entry  sheet,  as  well  as  the  weights  associated  with  each  measure  on 
the  Measure  Comments  and  Weights  sheet.  The  different  perspectives  are  defined  by  creating 
copies  of  the  first  eight  rows  of  the  Data  Entry  sheet,  keeping  measures  and  submeasures  in 
the  same  order. 

1.  Each  block  of  eight  rows  has  a  header  row  (highlighted  in  yellow).  The  first  entry  in  the 
header  row  should  contain  the  name  of  the  perspective  case  defined  below  it. 

2.  The  weight  of  each  measure  on  the  Measure  Comments  and  Weights  sheet  should  be  en¬ 
tered  in  a  cell  in  the  header  row  above  any  submeasure  of  that  measure. 

3.  Each  block  should  be  filled  with  the  desired  weights,  threshold  and  goal  values,  and  effec¬ 
tiveness  parameters  for  a  submeasure. 

The  header  row  separates  consecutive  perspective  cases,  so  the  completed  Perspective 
Cases  sheet  should  have  9P  rows  and  S  +  1  columns  filled,  where  P  is  the  number  of  perspec¬ 
tive  cases  and  S  is  the  number  of  submeasures.  Although  the  template  gives  the  row  headings 
for  four  perspective  cases,  more  perspectives  can  be  defined  below  these  cases. 
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Filling  in  the  Measure  Comments  and  Weights  Sheet 

Starting  in  row  2,  put  the  name  of  each  measure  in  the  first  column,  a  brief  comment  (which 
appears  when  the  cursor  moves  over  the  name  of  the  measure  in  the  Summary  sheet)  in  the 
second  column,  and  the  weight  of  the  measure  for  effectiveness  calculations  in  the  third  col¬ 
umn. 


Filling  in  the  IOC  to  Cost  Data  Sheet 

Although  this  sheet  is  not  required  for  PAT-MD  to  run,  it  does  allow  the  quick  generation 
of  RDT&E  annual  investments  for  each  BMDS  component  by  defining  the  desired  IOC 
date  for  each  ESC  for  each  investment  option.  The  sheet  consists  of  two  blocks,  one  con¬ 
taining  the  current  annual  RDT&E  investment  for  each  BMDS  component  (this  also  as¬ 
sumes  that  RDT&E  for  the  BMDS  components  is  completed  in  the  time  frame  under  con¬ 
sideration)  and  the  other  containing  the  names  of  investment  options  and  the  desired  IOC 
date  for  each  ESC.  These  blocks  and  their  arrangement  are  shown  in  Figure  B.2. 

1.  For  the  first  block,  the  first  row  (starting  in  column  B)  lists  the  years  in  the  time  frame 
under  consideration  in  consecutive  columns. 

2.  The  first  column  (starting  in  row  2)  lists  the  BMDS  components  (in  the  same  order  in 
which  they  appear  on  the  Cost  Data  sheet). 

3.  The  entries  in  the  table  are  the  RDT&E  investments  for  each  component  in  each  year  to 
complete  the  component’s  RDT&E.  It  is  assumed  that  the  last  non-zero  entry  in  the  row 
is  the  year  in  which  the  RDT&E  for  the  component  is  complete. 

Figure  B.2 

Block  Structure  of  IOC  to  Cost  Data  Sheet 
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4.  In  the  second  block  (which  should  be  separated  from  the  first  block  by  a  single  row),  the 
first  row  (starting  in  column  B)  lists  the  ESGs  (in  the  same  order  in  which  they  appear  on 
the  ESG  Data  sheet) . 

5.  The  first  column  of  the  block  (starting  in  the  second  row)  lists  the  investment  options. 

6.  The  entries  in  the  table  are  the  desired  IOC  for  each  ESG  for  each  investment  option.  If 
an  ESG  is  not  a  part  of  an  investment  option,  the  cell  should  be  left  blank. 

7.  Clicking  on  the  Generate  Cost  Data  button  will  produce  on  the  IOC  to  Cost  Data  Output 
sheet  blocks  identical  to  the  ones  that  appear  in  the  Cost  Data  sheet.  One  block  will  be 
generated  for  each  investment  option,  and  the  RDT&E  streams  are  such  that  every  com¬ 
ponent  is  done  on  time  for  each  ESG  to  be  completed  by  the  desired  IOC  date. 

Filling  in  the  Summary  and  Details  Sheets 

There  are  three  main  displays  on  the  Summary  sheet:  the  nine-column  color-coded  measure- 

summary  table,  four  columns  containing  values  of  individual  submeasures,  and  nine  columns 

of  cost  data. 

1.  Starting  in  row  3,  enter  the  names  of  the  investment  options  that  are  to  be  displayed  on 
the  Summary  sheet  (as  well  as  all  other  output  sheets)  in  the  first  column.  It  is  not  neces¬ 
sary  to  include  all  of  the  investment  options  on  the  Summary  sheet,  nor  is  it  necessary  for 
the  investment  options  to  appear  in  the  same  order  as  they  do  on  the  Data  Entry  sheet. 

2.  To  display  a  measure  in  the  measure-summary  table,  enter  the  name  of  the  measure  in 
the  first  row  above  the  column  in  which  it  is  to  be  displayed.  The  measures  do  not  all 
need  to  appear  on  the  Summary  sheet,  nor  must  they  appear  in  the  same  order  in  which 
they  appear  on  the  Data  Entry  sheet. 

3.  To  make  a  submeasure  appear  in  one  of  the  submeasure  columns,  enter  in  the  first  row 
the  name  of  the  measure  that  the  submeasure  is  associated  with,  followed  by  two  colons 
(::),  followed  by  the  name  of  the  submeasure  (e.g.,  A::B  would  access  submeasure  B  of 
measure  A) . 

4.  To  make  a  cost  metric  appear  in  the  cost-data  columns,  the  column  heading  should  have 
the  format  type  cost:  year-range ,  where  type  is  either  RDT&E,  Deployment,  or  Total 
(RDT&E  +  deployment)  and  year-range  is  either  a  single  year  in  the  time  frame  (e.g., 
2005)  or  a  start  year  and  an  end  year  separated  by  a  dash  (e.g.,  2005-2010).  A  sample 
cost-column  heading  would  be  “RDT&E  Cost:  2005-2010.” 

5.  Click  the  Populate  Scroll  Bars  button.  This  will  populate  all  of  the  dropdown  menus  used 
in  all  of  the  output  sheets. 

6.  Click  on  the  Modify  Summary  button.  This  will  fill  in  all  of  the  information  on  the 
Summary  sheet. 

7.  To  display  the  details  of  a  measure  that  is  a  column  on  the  Summary  sheet,  click  on  the 
Detail  button  for  that  column.  Similarly,  clicking  the  Related  Details  button  under  a 
submeasure  column  will  display  a  Details  sheet  with  all  of  the  submeasures  that  share  the 
measure  of  the  submeasure  in  that  column. 

8.  When  the  Details  sheet  is  open,  changing  the  measure  name  in  cell  B1  and  clicking  on 
the  Get  Details  button  will  display  all  of  the  submeasures  of  that  measure. 
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An  Example  of  Building  a  Portfolio  View 

Figures  B.3  through  B.10  present  a  template  for  a  very  simple  example.  The  example  has  two 
investment  options,  two  measures  (each  consisting  of  two  submeasures),  three  BMDS  com¬ 
ponents,  and  two  ESGs. 

Figure  B.3 
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Figure  B.4 
Cost  Data  Sheet 
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Figure  B.5 
ESG  Data  Sheet 
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Figure  B.6 

Portion  of  Perspective  Cases  Sheet 
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Note  that  the  weights  on  the  measures  change  with  the  different  perspectives 
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Figure  B.7 

Measure  Comments  and  Weights  Sheet 
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Figure  B.8 

Truncated  Summary  Sheet  (Unfilled  and  Filled) 
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Figure  B.9 

Truncated  Summary  Sheet  (with  Results) 
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Figure  B.10 
Details  Sheet 
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Template  Builder  Sheet 

To  facilitate  the  setup  of  worksheets,  we  have  included  a  Template  Builder  sheet  (sketched  in 
Figure  B.l  1)  as  the  last  sheet  in  the  PAT-MD  workbook.  All  of  the  names  of  the  investment 
options,  measures  and  submeasures,  and  so  forth  can  be  entered  on  the  Template  Builder 
sheet.  The  first  and  last  years  of  the  time  frame  under  consideration  are  entered  in  the  first 
two  rows,  and  the  names  of  the  investment  options  are  entered  in  the  first  column.  The 
name  of  each  measure  is  entered  in  the  second  column  once  for  each  collection  of  submea- 
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Figure  B.11 

Template  Builder  Sheet 
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sures.  The  names  of  the  submeasures  associated  with  each  measure  are  entered  in  consecutive 
rows  in  the  third  column,  with  “High”  or  “Low”  entered  in  the  fourth  column  to  denote 
whether  high  or  low  values  are  desired  for  the  submeasure.  The  names  of  the  BMDS  com¬ 
ponents  are  entered  in  the  fifth  column,  and  the  names  of  the  ESGs  are  entered  in  the  sixth 
column. 

Every  time  a  new  PAT-MD  template  is  built  (by  clicking  on  the  Build  Sheets  but¬ 
ton),  all  of  the  data  currently  on  all  of  the  worksheets  are  removed.  The  user  is  warned  that 
this  deletion  will  occur  and  has  the  option  of  canceling  before  proceeding  with  building  a 
new  PAT-MD  template.  The  row  and  column  labels  for  all  of  the  tables  on  all  of  the  sheets 
and  the  entries  for  all  of  the  dropdown  menus  are  generated,  but  the  data  must  still  be  en¬ 
tered  by  other  means. 
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